2024-03-24T11:35:42.229 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T11:35:42.229 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938 --owner scheduled_teuthology@teuthology --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} --name teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938/orig.config.yaml 2024-03-24T11:35:42.258 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:35:42.338 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7617938' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 8903 sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 sleep_before_teardown: 0 subset: 44/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 targets: smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBnNasDdYlxjP264AtAYghJBzdsnLAX8t+0AKvqGcs9IJBfqj2ipsYsTC1yzPnbJWWY1XZStiXWRA2nqrQnWClg= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBImy5hzhgK6dxJSSTzKRdpyWRTgeFmIeqf/BpTsHkd2ZIF/ariilZrnGUcxh32fQLSyW2vs2antgV7QBRpBvXYQ= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdWjCr8lwue03NKbJADyVIQ0h7TnSC0QizrEu49mmF4Le2cv/0dpMzkbE686tHQp+4FiWWJO9GrNsStsk/evGk= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T11:35:42.339 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa; will attempt to use it 2024-03-24T11:35:42.339 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks 2024-03-24T11:35:42.339 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T11:35:42.341 INFO:teuthology.task.internal:Checking packages... 2024-03-24T11:35:42.365 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'db0330b1e4e2470d52b750e251e55a522b4f7d69' 2024-03-24T11:35:42.365 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:35:42.365 INFO:teuthology.packaging:ref: None 2024-03-24T11:35:42.365 INFO:teuthology.packaging:tag: None 2024-03-24T11:35:42.365 INFO:teuthology.packaging:branch: squid 2024-03-24T11:35:42.365 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:35:42.365 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-24T11:35:42.510 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1764.gf73f1dfd 2024-03-24T11:35:42.511 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T11:35:42.518 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T11:35:42.518 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T11:35:42.525 INFO:teuthology.task.internal:Saving configuration 2024-03-24T11:35:42.536 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T11:35:42.542 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:35:42.563 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:29:31.896685', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBnNasDdYlxjP264AtAYghJBzdsnLAX8t+0AKvqGcs9IJBfqj2ipsYsTC1yzPnbJWWY1XZStiXWRA2nqrQnWClg='} 2024-03-24T11:35:42.581 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:29:31.897924', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBImy5hzhgK6dxJSSTzKRdpyWRTgeFmIeqf/BpTsHkd2ZIF/ariilZrnGUcxh32fQLSyW2vs2antgV7QBRpBvXYQ='} 2024-03-24T11:35:42.600 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:29:31.895160', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdWjCr8lwue03NKbJADyVIQ0h7TnSC0QizrEu49mmF4Le2cv/0dpMzkbE686tHQp+4FiWWJO9GrNsStsk/evGk='} 2024-03-24T11:35:42.600 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T11:35:42.606 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-24T11:35:42.607 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-24T11:35:42.607 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-24T11:35:42.607 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T11:35:42.699 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbbe0282f70>, signals=[15]) 2024-03-24T11:35:42.699 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T11:35:42.706 INFO:teuthology.task.internal:Opening connections... 2024-03-24T11:35:42.706 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2024-03-24T11:35:42.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:35:42.798 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-03-24T11:35:42.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:35:42.888 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-03-24T11:35:42.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:35:42.975 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T11:35:42.981 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2024-03-24T11:35:43.000 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2024-03-24T11:35:43.000 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2024-03-24T11:35:43.056 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="CentOS Stream" 2024-03-24T11:35:43.056 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="9" 2024-03-24T11:35:43.056 INFO:teuthology.orchestra.run.smithi130.stdout:ID="centos" 2024-03-24T11:35:43.056 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:35:43.056 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="9" 2024-03-24T11:35:43.056 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:35:43.056 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:35:43.057 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2024-03-24T11:35:43.057 INFO:teuthology.orchestra.run.smithi130.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:35:43.057 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:35:43.057 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:35:43.057 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:35:43.057 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:35:43.057 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:35:43.058 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2024-03-24T11:35:43.100 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-03-24T11:35:43.118 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-03-24T11:35:43.119 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-03-24T11:35:43.176 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-03-24T11:35:43.176 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-03-24T11:35:43.176 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-03-24T11:35:43.176 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:35:43.176 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-03-24T11:35:43.176 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:35:43.176 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:35:43.176 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-03-24T11:35:43.177 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:35:43.177 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:35:43.177 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:35:43.177 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:35:43.177 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:35:43.177 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:35:43.177 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-03-24T11:35:43.207 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-03-24T11:35:43.225 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-03-24T11:35:43.226 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-03-24T11:35:43.282 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-03-24T11:35:43.282 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-03-24T11:35:43.282 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-03-24T11:35:43.282 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:35:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-03-24T11:35:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:35:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:35:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-03-24T11:35:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:35:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:35:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:35:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:35:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:35:43.283 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:35:43.283 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-03-24T11:35:43.307 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T11:35:43.317 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T11:35:43.325 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T11:35:43.325 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:35:43.328 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:35:43.330 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:35:43.346 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T11:35:43.351 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T11:35:43.352 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:35:43.385 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:35:43.387 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:35:43.508 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T11:35:43.591 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T11:35:43.606 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T11:35:43.606 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T11:35:43.607 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:35:43.607 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:35:43.608 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:35:43.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:43.625 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-03-24T11:35:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:43.627 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-03-24T11:35:43.627 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-03-24T11:35:43.682 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:35:43.683 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.14.0-361.el9.x86_64 2024-03-24T11:35:43.683 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum install -y kernel 2024-03-24T11:35:43.684 INFO:teuthology.orchestra.run.smithi130.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:35:43.684 INFO:teuthology.task.kernel:Running kernel on smithi130: 5.14.0-361.el9.x86_64 2024-03-24T11:35:43.684 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum install -y kernel 2024-03-24T11:35:43.685 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:35:43.686 INFO:teuthology.task.kernel:Running kernel on smithi189: 5.14.0-361.el9.x86_64 2024-03-24T11:35:43.686 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum install -y kernel 2024-03-24T11:35:45.806 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T11:35:45.841 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-24T11:35:45.922 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-24T11:35:47.415 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-24T11:35:47.425 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-24T11:35:47.581 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-24T11:35:48.689 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-03-24T11:35:49.640 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.0 MB 00:01 2024-03-24T11:35:50.365 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-03-24T11:35:50.713 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.0 MB 00:02 2024-03-24T11:35:51.610 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-24T11:35:55.614 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 19 MB 00:03 2024-03-24T11:35:56.561 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-24T11:35:58.193 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-24T11:35:58.348 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.1 MB 00:04 2024-03-24T11:35:59.548 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-24T11:35:59.857 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-03-24T11:36:00.403 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 21 MB 00:02 2024-03-24T11:36:01.473 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 16 kB 00:00 2024-03-24T11:36:01.811 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-24T11:36:04.052 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 21 MB 00:02 2024-03-24T11:36:05.327 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-24T11:36:06.683 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-24T11:36:07.443 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:36:07.443 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:36:07.540 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-24T11:36:07.546 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-24T11:36:07.546 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-03-24T11:36:07.546 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-24T11:36:07.547 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-24T11:36:07.547 INFO:teuthology.orchestra.run.smithi130.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:36:07.547 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-24T11:36:07.547 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:36:07.547 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:36:07.547 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:36:07.547 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:36:07.547 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-24T11:36:07.547 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-24T11:36:07.547 INFO:teuthology.orchestra.run.smithi130.stdout:Install 4 Packages 2024-03-24T11:36:07.547 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:36:07.548 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 76 M 2024-03-24T11:36:07.548 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 123 M 2024-03-24T11:36:07.548 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-24T11:36:07.879 INFO:teuthology.orchestra.run.smithi130.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 258 kB/s | 44 kB 00:00 2024-03-24T11:36:08.798 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:36:08.798 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:36:08.898 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout:Install 4 Packages 2024-03-24T11:36:08.904 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-24T11:36:08.905 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 76 M 2024-03-24T11:36:08.905 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 123 M 2024-03-24T11:36:08.905 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-24T11:36:09.074 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-24T11:36:09.076 INFO:teuthology.orchestra.run.smithi174.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 343 kB/s | 44 kB 00:00 2024-03-24T11:36:09.754 INFO:teuthology.orchestra.run.smithi130.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 7.3 MB/s | 15 MB 00:02 2024-03-24T11:36:10.537 INFO:teuthology.orchestra.run.smithi130.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 10 MB/s | 27 MB 00:02 2024-03-24T11:36:10.826 INFO:teuthology.orchestra.run.smithi174.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 7.9 MB/s | 15 MB 00:01 2024-03-24T11:36:11.004 INFO:teuthology.orchestra.run.smithi130.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 10 MB/s | 33 MB 00:03 2024-03-24T11:36:11.005 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:36:11.005 INFO:teuthology.orchestra.run.smithi130.stdout:Total 22 MB/s | 76 MB 00:03 2024-03-24T11:36:11.164 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:36:11.165 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:36:11.261 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-24T11:36:11.270 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:36:11.270 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout:Install 4 Packages 2024-03-24T11:36:11.271 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:36:11.272 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 76 M 2024-03-24T11:36:11.272 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 123 M 2024-03-24T11:36:11.272 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-24T11:36:11.517 INFO:teuthology.orchestra.run.smithi189.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 356 kB/s | 44 kB 00:00 2024-03-24T11:36:11.670 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-24T11:36:12.026 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-24T11:36:12.026 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-24T11:36:13.015 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-24T11:36:13.016 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-24T11:36:13.434 INFO:teuthology.orchestra.run.smithi174.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 7.5 MB/s | 33 MB 00:04 2024-03-24T11:36:14.068 INFO:teuthology.orchestra.run.smithi174.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 5.5 MB/s | 27 MB 00:04 2024-03-24T11:36:14.068 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:36:14.068 INFO:teuthology.orchestra.run.smithi174.stdout:Total 15 MB/s | 76 MB 00:05 2024-03-24T11:36:14.733 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-24T11:36:15.093 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-24T11:36:15.093 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-24T11:36:15.476 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-24T11:36:16.083 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:36:16.105 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-24T11:36:16.106 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-24T11:36:16.172 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:36:18.594 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-24T11:36:18.854 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:36:19.000 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:36:19.231 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:36:19.317 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:36:20.058 INFO:teuthology.orchestra.run.smithi189.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 1.7 MB/s | 15 MB 00:08 2024-03-24T11:36:22.026 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:36:22.153 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:36:23.698 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:36:23.810 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:26.893 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:36:26.928 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:28.558 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:31.667 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:35.123 INFO:teuthology.orchestra.run.smithi189.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.2 MB/s | 27 MB 00:23 2024-03-24T11:36:40.573 INFO:teuthology.orchestra.run.smithi189.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.1 MB/s | 33 MB 00:29 2024-03-24T11:36:40.573 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:36:40.573 INFO:teuthology.orchestra.run.smithi189.stdout:Total 2.6 MB/s | 76 MB 00:29 2024-03-24T11:36:41.233 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-24T11:36:41.791 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-24T11:36:41.791 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-24T11:36:42.767 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-24T11:36:42.768 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-24T11:36:45.238 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-24T11:36:45.919 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:36:46.006 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:36:48.779 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:36:48.916 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:36:53.602 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:36:53.643 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:57.777 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:57.777 INFO:teuthology.orchestra.run.smithi130.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:36:57.777 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:36:57.821 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:58.230 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:58.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:36:58.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:36:58.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:36:58.363 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:58.655 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:36:58.655 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:36:58.656 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-24T11:36:58.656 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:36:58.656 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:36:58.656 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:36:58.656 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:36:58.656 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:36:58.656 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-24T11:36:58.972 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:36:59.108 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:36:59.109 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:36:59.109 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:36:59.109 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:36:59.109 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:36:59.109 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T11:36:59.109 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi130.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:36:59.109 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:36:59.189 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:36:59.189 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:36:59.189 DEBUG:teuthology.orchestra.run.smithi130:> sudo rpm -qi grub2-tools 2024-03-24T11:36:59.234 INFO:teuthology.orchestra.run.smithi130.stdout:Name : grub2-tools 2024-03-24T11:36:59.235 INFO:teuthology.orchestra.run.smithi130.stdout:Epoch : 1 2024-03-24T11:36:59.235 INFO:teuthology.orchestra.run.smithi130.stdout:Version : 2.06 2024-03-24T11:36:59.235 INFO:teuthology.orchestra.run.smithi130.stdout:Release : 68.el9 2024-03-24T11:36:59.235 INFO:teuthology.orchestra.run.smithi130.stdout:Architecture: x86_64 2024-03-24T11:36:59.235 INFO:teuthology.orchestra.run.smithi130.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:36:59.235 INFO:teuthology.orchestra.run.smithi130.stdout:Group : Unspecified 2024-03-24T11:36:59.235 INFO:teuthology.orchestra.run.smithi130.stdout:Size : 8264254 2024-03-24T11:36:59.235 INFO:teuthology.orchestra.run.smithi130.stdout:License : GPLv3+ 2024-03-24T11:36:59.235 INFO:teuthology.orchestra.run.smithi130.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:36:59.235 INFO:teuthology.orchestra.run.smithi130.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:36:59.235 INFO:teuthology.orchestra.run.smithi130.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout:Packager : builder@centos.org 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout:Vendor : CentOS 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout:Summary : Support tools for GRUB. 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout:Description : 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout:hardware devices. 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:36:59.236 INFO:teuthology.orchestra.run.smithi130.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:36:59.237 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:36:59.237 INFO:teuthology.task.kernel:Updating grub on smithi130 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:36:59.238 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:36:59.821 INFO:teuthology.orchestra.run.smithi130.stderr:Generating grub configuration file ... 2024-03-24T11:37:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:37:00.871 INFO:teuthology.orchestra.run.smithi174.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:37:00.871 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-24T11:37:00.915 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:37:01.153 INFO:teuthology.orchestra.run.smithi130.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:37:01.192 INFO:teuthology.orchestra.run.smithi130.stderr:done 2024-03-24T11:37:01.194 DEBUG:teuthology.orchestra.run.smithi130:> mktemp 2024-03-24T11:37:01.210 INFO:teuthology.orchestra.run.smithi130.stdout:/tmp/tmp.MjGrQLAzHL 2024-03-24T11:37:01.210 DEBUG:teuthology.orchestra.run.smithi130:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MjGrQLAzHL 2024-03-24T11:37:01.278 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0666 /tmp/tmp.MjGrQLAzHL 2024-03-24T11:37:01.351 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:37:01.351 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:37:01.351 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:37:01.351 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:37:01.426 DEBUG:teuthology.orchestra.remote:smithi130:/tmp/tmp.MjGrQLAzHL is 6KB 2024-03-24T11:37:01.445 DEBUG:teuthology.orchestra.run.smithi130:> rm -fr /tmp/tmp.MjGrQLAzHL 2024-03-24T11:37:01.459 DEBUG:teuthology.orchestra.run.smithi130:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:37:01.524 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:37:01.524 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:37:01.524 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:37:01.524 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:37:01.525 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:37:01.693 DEBUG:teuthology.orchestra.run.smithi130:> sudo shutdown -r now 2024-03-24T11:37:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:37:01.726 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-24T11:37:01.727 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-24T11:37:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:37:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:37:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:37:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:37:01.727 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-24T11:37:01.727 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-24T11:37:02.066 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:37:02.203 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:37:02.203 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:37:02.203 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:37:02.203 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:37:02.203 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:37:02.203 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-24T11:37:02.203 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:37:02.203 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:37:02.282 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:37:02.282 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:37:02.283 DEBUG:teuthology.orchestra.run.smithi174:> sudo rpm -qi grub2-tools 2024-03-24T11:37:02.316 INFO:teuthology.orchestra.run.smithi174.stdout:Name : grub2-tools 2024-03-24T11:37:02.316 INFO:teuthology.orchestra.run.smithi174.stdout:Epoch : 1 2024-03-24T11:37:02.316 INFO:teuthology.orchestra.run.smithi174.stdout:Version : 2.06 2024-03-24T11:37:02.316 INFO:teuthology.orchestra.run.smithi174.stdout:Release : 68.el9 2024-03-24T11:37:02.316 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: x86_64 2024-03-24T11:37:02.316 INFO:teuthology.orchestra.run.smithi174.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:37:02.316 INFO:teuthology.orchestra.run.smithi174.stdout:Group : Unspecified 2024-03-24T11:37:02.316 INFO:teuthology.orchestra.run.smithi174.stdout:Size : 8264254 2024-03-24T11:37:02.316 INFO:teuthology.orchestra.run.smithi174.stdout:License : GPLv3+ 2024-03-24T11:37:02.316 INFO:teuthology.orchestra.run.smithi174.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:Packager : builder@centos.org 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:Vendor : CentOS 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:Summary : Support tools for GRUB. 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:Description : 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:hardware devices. 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-24T11:37:02.317 INFO:teuthology.orchestra.run.smithi174.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:37:02.318 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:37:02.318 INFO:teuthology.task.kernel:Updating grub on smithi174 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:37:02.318 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:37:02.908 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2024-03-24T11:37:04.339 INFO:teuthology.orchestra.run.smithi174.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:37:04.365 INFO:teuthology.orchestra.run.smithi174.stderr:done 2024-03-24T11:37:04.367 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-03-24T11:37:04.385 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.grUPK0ShcD 2024-03-24T11:37:04.385 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.grUPK0ShcD 2024-03-24T11:37:04.455 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.grUPK0ShcD 2024-03-24T11:37:04.590 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.grUPK0ShcD is 6KB 2024-03-24T11:37:04.638 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.grUPK0ShcD 2024-03-24T11:37:04.653 DEBUG:teuthology.orchestra.run.smithi174:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:37:04.717 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:37:04.717 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:37:04.717 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:37:04.717 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:37:04.718 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:37:04.874 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2024-03-24T11:37:27.402 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:37:27.403 INFO:teuthology.orchestra.run.smithi189.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:37:27.403 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:37:27.446 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:37:27.875 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:37:27.875 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:37:27.875 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:37:27.875 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:37:28.277 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:37:28.277 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:37:28.277 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-24T11:37:28.277 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:37:28.277 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:37:28.277 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:37:28.277 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:37:28.277 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:37:28.277 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-24T11:37:28.633 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:37:28.729 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:37:28.730 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:37:28.730 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:37:28.730 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:37:28.730 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:37:28.730 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-24T11:37:28.730 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:37:28.730 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:37:28.810 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:37:28.810 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:37:28.810 DEBUG:teuthology.orchestra.run.smithi189:> sudo rpm -qi grub2-tools 2024-03-24T11:37:28.850 INFO:teuthology.orchestra.run.smithi189.stdout:Name : grub2-tools 2024-03-24T11:37:28.850 INFO:teuthology.orchestra.run.smithi189.stdout:Epoch : 1 2024-03-24T11:37:28.850 INFO:teuthology.orchestra.run.smithi189.stdout:Version : 2.06 2024-03-24T11:37:28.850 INFO:teuthology.orchestra.run.smithi189.stdout:Release : 68.el9 2024-03-24T11:37:28.850 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: x86_64 2024-03-24T11:37:28.850 INFO:teuthology.orchestra.run.smithi189.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:37:28.850 INFO:teuthology.orchestra.run.smithi189.stdout:Group : Unspecified 2024-03-24T11:37:28.850 INFO:teuthology.orchestra.run.smithi189.stdout:Size : 8264254 2024-03-24T11:37:28.850 INFO:teuthology.orchestra.run.smithi189.stdout:License : GPLv3+ 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:Packager : builder@centos.org 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:Vendor : CentOS 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:Summary : Support tools for GRUB. 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:Description : 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:hardware devices. 2024-03-24T11:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:37:28.852 INFO:teuthology.orchestra.run.smithi189.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:37:28.852 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:37:28.852 INFO:teuthology.task.kernel:Updating grub on smithi189 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:37:28.852 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:37:29.437 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2024-03-24T11:37:30.779 INFO:teuthology.orchestra.run.smithi189.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:37:30.799 INFO:teuthology.orchestra.run.smithi189.stderr:done 2024-03-24T11:37:30.801 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-03-24T11:37:30.816 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.JhOFOBrgXN 2024-03-24T11:37:30.816 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JhOFOBrgXN 2024-03-24T11:37:30.884 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.JhOFOBrgXN 2024-03-24T11:37:31.018 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.JhOFOBrgXN is 6KB 2024-03-24T11:37:31.066 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.JhOFOBrgXN 2024-03-24T11:37:31.080 DEBUG:teuthology.orchestra.run.smithi189:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:37:31.145 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:37:31.145 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:37:31.145 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:37:31.145 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:37:31.147 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:37:31.316 DEBUG:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2024-03-24T11:37:31.696 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T11:37:31.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-24T11:37:31.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:37:34.879 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T11:37:34.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-24T11:37:34.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:37:50.114 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2024-03-24T11:37:53.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-03-24T11:37:59.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-24T11:37:59.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:01.322 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-24T11:38:01.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:38:01.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:02.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-24T11:38:02.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:05.506 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-03-24T11:38:17.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-24T11:38:17.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:19.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-03-24T11:38:28.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:38:28.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:31.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-03-24T11:38:43.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:38:43.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:38:49.271 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-03-24T11:38:49.751 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-24T11:38:49.752 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:38:49.752 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-03-24T11:38:49.765 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:38:49.766 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:38:49.766 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:38:49.766 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:38:50.766 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T11:38:50.767 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:38:50.849 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2024-03-24T11:38:50.877 DEBUG:teuthology.parallel:result is None 2024-03-24T11:38:59.121 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T11:39:11.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-24T11:39:11.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:39:11.470 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-03-24T11:39:11.859 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-24T11:39:11.859 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:39:11.859 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-03-24T11:39:11.874 INFO:teuthology.orchestra.run.smithi130.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:39:11.874 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:39:11.875 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:39:11.875 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:39:12.875 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T11:39:12.876 DEBUG:teuthology.orchestra.run.smithi130:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:39:12.946 INFO:teuthology.orchestra.run.smithi130.stdout:ttyS1 2024-03-24T11:39:12.987 DEBUG:teuthology.parallel:result is None 2024-03-24T11:39:43.886 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T11:39:58.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:39:58.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:39:59.129 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-03-24T11:39:59.521 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:39:59.521 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:39:59.522 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-03-24T11:39:59.537 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:39:59.537 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:39:59.538 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:39:59.538 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:40:00.538 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-24T11:40:00.539 DEBUG:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:40:00.616 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2024-03-24T11:40:00.661 DEBUG:teuthology.parallel:result is None 2024-03-24T11:40:00.662 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T11:40:00.670 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T11:40:00.670 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:40:00.673 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:40:00.676 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:40:00.693 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T11:40:00.704 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T11:40:00.718 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T11:40:00.719 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:40:00.732 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:40:00.735 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:40:00.814 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T11:40:00.828 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T11:40:00.828 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:40:00.834 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:40:00.837 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:40:00.875 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:40:00.880 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:40:00.882 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:40:00.889 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:40:00.895 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:40:00.897 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:40:00.898 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T11:40:00.907 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T11:40:00.907 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:40:00.910 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:40:00.912 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:40:00.960 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T11:40:00.980 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T11:40:00.980 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:40:00.983 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:40:00.985 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:40:01.015 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:40:01.080 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:40:01.133 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:40:01.215 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:40:01.258 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:40:01.259 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:40:01.325 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:40:01.371 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:40:01.425 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:40:01.510 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:40:01.554 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-24T11:40:01.554 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:40:01.623 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:40:01.671 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:40:01.733 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:40:01.777 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:40:01.821 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:40:01.821 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:40:01.891 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2024-03-24T11:40:01.893 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-03-24T11:40:01.896 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-03-24T11:40:01.940 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:40:01.941 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:40:01.975 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:40:02.382 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T11:40:02.390 INFO:teuthology.task.internal:Starting timer... 2024-03-24T11:40:02.390 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T11:40:02.407 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T11:40:02.420 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-24T11:40:02.421 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2024-03-24T11:40:02.488 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2024-03-24T11:40:02.490 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-03-24T11:40:02.548 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-03-24T11:40:02.551 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-03-24T11:40:02.602 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-03-24T11:40:02.604 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T11:40:02.604 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2024-03-24T11:40:02.639 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2024-03-24T11:40:02.639 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-03-24T11:40:02.659 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-03-24T11:40:02.659 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-03-24T11:40:02.692 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-03-24T11:40:02.692 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi130.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-03-24T11:40:02.693 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:40:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:02.722 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:40:02.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:02.751 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:40:02.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:02.779 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T11:40:02.779 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:40:02.815 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:40:02.870 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:40:02.910 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T11:40:02.921 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T11:40:02.942 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T11:40:02.944 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi130.front.sepia.ceph.com,smithi174.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T11:44:24.239 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi130.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-03-24T11:44:24.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-24T11:44:24.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:44:24.330 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-03-24T11:44:24.410 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-24T11:44:24.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-24T11:44:24.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:44:24.505 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-03-24T11:44:24.589 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-03-24T11:44:24.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:44:24.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:44:24.680 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-03-24T11:44:24.761 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:44:24.762 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T11:44:24.772 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T11:44:24.773 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:44:24.773 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:44:24.776 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:44:24.776 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:44:24.778 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:44:24.778 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:44:24.806 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:44:24.809 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:44:24.823 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:44:24.827 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:44:24.832 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:44:24.849 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:44:24.853 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2024-03-24T11:44:24.857 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-03-24T11:44:24.866 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2024-03-24T11:44:24.871 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-03-24T11:44:24.879 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-03-24T11:44:24.883 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:44:24.888 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:44:24.892 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-03-24T11:44:24.899 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:44:24.904 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:44:24.909 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:44:24.925 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:44:24.948 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:44:24.952 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:44:24.952 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-03-24T11:44:24.952 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.952 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.952 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.952 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.955 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:44:24.958 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:44:24.958 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-03-24T11:44:24.959 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.959 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.959 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.959 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.985 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:44:24.989 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:44:24.989 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-24T11:44:24.989 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.990 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.990 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.990 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:44:24.990 INFO:teuthology.run_tasks:Running task pexec... 2024-03-24T11:44:25.000 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-24T11:44:25.000 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi130.front.sepia.ceph.com 2024-03-24T11:44:25.000 DEBUG:teuthology.orchestra.run.smithi130:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T11:44:25.001 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi174.front.sepia.ceph.com 2024-03-24T11:44:25.001 DEBUG:teuthology.orchestra.run.smithi174:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T11:44:25.001 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi189.front.sepia.ceph.com 2024-03-24T11:44:25.001 DEBUG:teuthology.orchestra.run.smithi189:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T11:44:25.585 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:48 ago on Sun 24 Mar 2024 11:43:37 AM UTC. 2024-03-24T11:44:25.586 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:55 ago on Sun 24 Mar 2024 11:43:30 AM UTC. 2024-03-24T11:44:25.613 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:50 ago on Sun 24 Mar 2024 11:43:35 AM UTC. 2024-03-24T11:44:25.732 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-24T11:44:25.734 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-24T11:44:25.734 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-03-24T11:44:25.734 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-24T11:44:25.734 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-24T11:44:25.734 INFO:teuthology.orchestra.run.smithi130.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T11:44:25.734 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:44:25.734 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-24T11:44:25.735 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-24T11:44:25.735 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-24T11:44:25.735 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:44:25.736 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-03-24T11:44:25.737 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-24T11:44:25.737 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-03-24T11:44:25.737 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-24T11:44:25.738 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-03-24T11:44:25.738 INFO:teuthology.orchestra.run.smithi174.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T11:44:25.738 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-24T11:44:25.738 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-03-24T11:44:25.738 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-03-24T11:44:25.738 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-03-24T11:44:25.738 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-24T11:44:25.746 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 3.1 M 2024-03-24T11:44:25.746 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 9.8 M 2024-03-24T11:44:25.746 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-24T11:44:25.760 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-24T11:44:25.761 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:44:25.762 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-03-24T11:44:25.762 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:44:25.762 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-24T11:44:25.762 INFO:teuthology.orchestra.run.smithi189.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T11:44:25.762 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:44:25.762 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-24T11:44:25.762 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:44:25.762 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-03-24T11:44:25.762 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:44:25.763 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 3.1 M 2024-03-24T11:44:25.763 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 9.8 M 2024-03-24T11:44:25.763 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-03-24T11:44:25.776 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 3.1 M 2024-03-24T11:44:25.776 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 9.8 M 2024-03-24T11:44:25.776 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-24T11:44:26.021 INFO:teuthology.orchestra.run.smithi130.stdout:runc-1.1.12-2.el9.x86_64.rpm 13 MB/s | 3.1 MB 00:00 2024-03-24T11:44:26.022 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:26.022 INFO:teuthology.orchestra.run.smithi130.stdout:Total 11 MB/s | 3.1 MB 00:00 2024-03-24T11:44:26.045 INFO:teuthology.orchestra.run.smithi189.stdout:runc-1.1.12-2.el9.x86_64.rpm 21 MB/s | 3.1 MB 00:00 2024-03-24T11:44:26.045 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:26.045 INFO:teuthology.orchestra.run.smithi189.stdout:Total 12 MB/s | 3.1 MB 00:00 2024-03-24T11:44:26.057 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-24T11:44:26.063 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-24T11:44:26.063 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-24T11:44:26.080 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-24T11:44:26.086 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-24T11:44:26.086 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-24T11:44:26.141 INFO:teuthology.orchestra.run.smithi174.stdout:runc-1.1.12-2.el9.x86_64.rpm 9.3 MB/s | 3.1 MB 00:00 2024-03-24T11:44:26.142 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:44:26.142 INFO:teuthology.orchestra.run.smithi174.stdout:Total 8.2 MB/s | 3.1 MB 00:00 2024-03-24T11:44:26.143 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-24T11:44:26.143 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-24T11:44:26.165 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-24T11:44:26.165 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-24T11:44:26.179 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-03-24T11:44:26.185 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-03-24T11:44:26.185 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-03-24T11:44:26.265 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-03-24T11:44:26.266 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-03-24T11:44:26.351 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-24T11:44:26.384 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-24T11:44:26.433 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:44:26.473 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:44:26.491 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-03-24T11:44:26.586 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:44:26.971 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:44:27.070 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:44:27.102 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:44:27.285 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:44:27.285 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:44:27.285 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-24T11:44:27.285 INFO:teuthology.orchestra.run.smithi130.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T11:44:27.285 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:44:27.285 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-24T11:44:27.398 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:44:27.398 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:44:27.398 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-24T11:44:27.398 INFO:teuthology.orchestra.run.smithi189.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T11:44:27.398 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:44:27.398 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-24T11:44:27.416 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:44:27.416 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-24T11:44:27.416 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-03-24T11:44:27.416 INFO:teuthology.orchestra.run.smithi174.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T11:44:27.417 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-03-24T11:44:27.417 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-03-24T11:44:27.517 DEBUG:teuthology.parallel:result is None 2024-03-24T11:44:27.637 DEBUG:teuthology.parallel:result is None 2024-03-24T11:44:27.679 DEBUG:teuthology.parallel:result is None 2024-03-24T11:44:27.680 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-24T11:44:27.693 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-24T11:44:27.694 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:44:27.694 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:44:27.709 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:44:27.710 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_1 2024-03-24T11:44:27.765 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:44:27.765 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:27.765 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-24T11:44:27.765 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:27.765 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:27.765 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-24 11:44:23.182106653 +0000 2024-03-24T11:44:27.765 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-24 11:44:02.065485061 +0000 2024-03-24T11:44:27.765 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-24 11:44:02.065485061 +0000 2024-03-24T11:44:27.766 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-24 11:44:02.065485061 +0000 2024-03-24T11:44:27.766 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:44:27.830 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-24T11:44:27.830 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-24T11:44:27.831 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000266054 s, 1.9 MB/s 2024-03-24T11:44:27.832 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:44:27.887 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_2 2024-03-24T11:44:27.942 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:44:27.942 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:27.942 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T11:44:27.942 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:27.942 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:27.942 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-24 11:44:23.182106653 +0000 2024-03-24T11:44:27.942 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-24 11:44:02.417478745 +0000 2024-03-24T11:44:27.942 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-24 11:44:02.417478745 +0000 2024-03-24T11:44:27.942 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-24 11:44:02.417478745 +0000 2024-03-24T11:44:27.943 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:44:28.007 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-24T11:44:28.008 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-24T11:44:28.008 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.00021207 s, 2.4 MB/s 2024-03-24T11:44:28.009 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:44:28.065 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_3 2024-03-24T11:44:28.121 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:44:28.121 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:28.121 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T11:44:28.121 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:28.121 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:28.121 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-24 11:44:23.182106653 +0000 2024-03-24T11:44:28.121 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-24 11:44:02.762472555 +0000 2024-03-24T11:44:28.122 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-24 11:44:02.762472555 +0000 2024-03-24T11:44:28.122 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-24 11:44:02.762472555 +0000 2024-03-24T11:44:28.122 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:44:28.186 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-24T11:44:28.186 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-24T11:44:28.186 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.00019996 s, 2.6 MB/s 2024-03-24T11:44:28.188 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:44:28.243 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_4 2024-03-24T11:44:28.298 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:44:28.298 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:28.298 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T11:44:28.298 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:28.298 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:28.298 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-24 11:44:23.182106653 +0000 2024-03-24T11:44:28.298 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-24 11:44:03.123466078 +0000 2024-03-24T11:44:28.298 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-24 11:44:03.123466078 +0000 2024-03-24T11:44:28.299 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-24 11:44:03.123466078 +0000 2024-03-24T11:44:28.299 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:44:28.364 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-24T11:44:28.364 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-24T11:44:28.364 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000194015 s, 2.6 MB/s 2024-03-24T11:44:28.365 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:44:28.420 DEBUG:teuthology.orchestra.run.smithi130:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-24T11:44:28.568 INFO:teuthology.orchestra.run.smithi130.stdout:loop 2024-03-24T11:44:28.569 INFO:tasks.nvme_loop:Connecting nvme_loop smithi130:/dev/vg_nvme/lv_1... 2024-03-24T11:44:28.570 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-24T11:44:28.607 INFO:teuthology.orchestra.run.smithi130.stdout:1 2024-03-24T11:44:28.644 INFO:teuthology.orchestra.run.smithi130.stdout:/dev/vg_nvme/lv_11 2024-03-24T11:44:28.673 INFO:tasks.nvme_loop:Connecting nvme_loop smithi130:/dev/vg_nvme/lv_2... 2024-03-24T11:44:28.674 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-24T11:44:28.752 INFO:teuthology.orchestra.run.smithi130.stdout:1 2024-03-24T11:44:28.789 INFO:teuthology.orchestra.run.smithi130.stdout:/dev/vg_nvme/lv_21 2024-03-24T11:44:28.817 INFO:tasks.nvme_loop:Connecting nvme_loop smithi130:/dev/vg_nvme/lv_3... 2024-03-24T11:44:28.818 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-24T11:44:28.895 INFO:teuthology.orchestra.run.smithi130.stdout:1 2024-03-24T11:44:28.932 INFO:teuthology.orchestra.run.smithi130.stdout:/dev/vg_nvme/lv_31 2024-03-24T11:44:28.960 INFO:tasks.nvme_loop:Connecting nvme_loop smithi130:/dev/vg_nvme/lv_4... 2024-03-24T11:44:28.961 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-24T11:44:29.038 INFO:teuthology.orchestra.run.smithi130.stdout:1 2024-03-24T11:44:29.075 INFO:teuthology.orchestra.run.smithi130.stdout:/dev/vg_nvme/lv_41 2024-03-24T11:44:29.103 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:44:29.103 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:44:29.157 DEBUG:teuthology.orchestra.run.smithi130:> sudo nvme list 2024-03-24T11:44:29.222 INFO:teuthology.orchestra.run.smithi130.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-24T11:44:29.223 INFO:teuthology.orchestra.run.smithi130.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-24T11:44:29.223 INFO:teuthology.orchestra.run.smithi130.stdout:/dev/nvme0n1 PHFT620500Q1400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-24T11:44:29.223 INFO:teuthology.orchestra.run.smithi130.stdout:/dev/nvme1n1 2fb49f481a09d25465fc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:29.223 INFO:teuthology.orchestra.run.smithi130.stdout:/dev/nvme2n1 ba58472c4149d39e9be8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:29.223 INFO:teuthology.orchestra.run.smithi130.stdout:/dev/nvme3n1 56a859b84f523c3bf4b3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:29.223 INFO:teuthology.orchestra.run.smithi130.stdout:/dev/nvme4n1 9194ecab66d47b0f253f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:29.224 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:44:29.224 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:44:29.224 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/scratch_devs 2024-03-24T11:44:29.291 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-24T11:44:29.291 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:44:29.310 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:44:29.310 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-03-24T11:44:29.366 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:44:29.367 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:29.367 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T11:44:29.367 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:29.367 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:29.367 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-24 11:44:23.243956341 +0000 2024-03-24T11:44:29.367 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-24 11:43:55.452315250 +0000 2024-03-24T11:44:29.367 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-24 11:43:55.452315250 +0000 2024-03-24T11:44:29.367 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-24 11:43:55.452315250 +0000 2024-03-24T11:44:29.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:44:29.435 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-24T11:44:29.435 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-24T11:44:29.435 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000280074 s, 1.8 MB/s 2024-03-24T11:44:29.437 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:44:29.494 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-03-24T11:44:29.549 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:44:29.549 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:29.550 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-24T11:44:29.550 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:29.550 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:29.550 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-24 11:44:23.243956341 +0000 2024-03-24T11:44:29.550 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-24 11:43:55.788310906 +0000 2024-03-24T11:44:29.550 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-24 11:43:55.788310906 +0000 2024-03-24T11:44:29.550 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-24 11:43:55.788310906 +0000 2024-03-24T11:44:29.550 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:44:29.617 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-24T11:44:29.617 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-24T11:44:29.617 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000235784 s, 2.2 MB/s 2024-03-24T11:44:29.618 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:44:29.675 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-03-24T11:44:29.731 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:44:29.731 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:29.732 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-24T11:44:29.732 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:29.732 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:29.732 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-24 11:44:23.243956341 +0000 2024-03-24T11:44:29.732 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-24 11:43:56.131306473 +0000 2024-03-24T11:44:29.732 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-24 11:43:56.131306473 +0000 2024-03-24T11:44:29.732 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-24 11:43:56.131306473 +0000 2024-03-24T11:44:29.732 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:44:29.797 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-24T11:44:29.797 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-24T11:44:29.797 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000167896 s, 3.0 MB/s 2024-03-24T11:44:29.799 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:44:29.854 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-03-24T11:44:29.909 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:44:29.910 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:29.910 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T11:44:29.910 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:29.910 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:29.910 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-24 11:44:23.243956341 +0000 2024-03-24T11:44:29.910 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-24 11:43:56.483301923 +0000 2024-03-24T11:44:29.910 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-24 11:43:56.483301923 +0000 2024-03-24T11:44:29.910 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-24 11:43:56.483301923 +0000 2024-03-24T11:44:29.910 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:44:29.975 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-24T11:44:29.975 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-24T11:44:29.975 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000193191 s, 2.7 MB/s 2024-03-24T11:44:29.976 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:44:30.032 DEBUG:teuthology.orchestra.run.smithi174:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-24T11:44:30.177 INFO:teuthology.orchestra.run.smithi174.stdout:loop 2024-03-24T11:44:30.178 INFO:tasks.nvme_loop:Connecting nvme_loop smithi174:/dev/vg_nvme/lv_1... 2024-03-24T11:44:30.178 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-24T11:44:30.216 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-03-24T11:44:30.254 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/vg_nvme/lv_11 2024-03-24T11:44:30.283 INFO:tasks.nvme_loop:Connecting nvme_loop smithi174:/dev/vg_nvme/lv_2... 2024-03-24T11:44:30.284 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-24T11:44:30.381 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-03-24T11:44:30.419 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/vg_nvme/lv_21 2024-03-24T11:44:30.449 INFO:tasks.nvme_loop:Connecting nvme_loop smithi174:/dev/vg_nvme/lv_3... 2024-03-24T11:44:30.449 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-24T11:44:30.500 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-03-24T11:44:30.538 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/vg_nvme/lv_31 2024-03-24T11:44:30.567 INFO:tasks.nvme_loop:Connecting nvme_loop smithi174:/dev/vg_nvme/lv_4... 2024-03-24T11:44:30.567 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-24T11:44:30.647 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-03-24T11:44:30.686 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/vg_nvme/lv_41 2024-03-24T11:44:30.716 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-24T11:44:30.716 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:44:30.772 DEBUG:teuthology.orchestra.run.smithi174:> sudo nvme list 2024-03-24T11:44:30.838 INFO:teuthology.orchestra.run.smithi174.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-24T11:44:30.838 INFO:teuthology.orchestra.run.smithi174.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-24T11:44:30.838 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/nvme0n1 PHFT62050074400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-24T11:44:30.838 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/nvme1n1 e085c5229b4668f067dc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:30.839 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/nvme2n1 fe993b0f2ead27ce2e37 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:30.839 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/nvme3n1 aee018fc9c94715c446f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:30.839 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/nvme4n1 b13ba8c559cf545299ce Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:30.840 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:44:30.840 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-24T11:44:30.840 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/scratch_devs 2024-03-24T11:44:30.907 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:44:30.907 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:44:30.926 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:44:30.926 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-03-24T11:44:30.983 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:44:30.983 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:30.983 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-24T11:44:30.983 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:30.983 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:30.983 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:44:23.259874011 +0000 2024-03-24T11:44:30.983 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:44:00.078327965 +0000 2024-03-24T11:44:30.983 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:44:00.078327965 +0000 2024-03-24T11:44:30.983 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:44:00.078327965 +0000 2024-03-24T11:44:30.984 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:44:31.052 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:44:31.052 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:44:31.052 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000278141 s, 1.8 MB/s 2024-03-24T11:44:31.054 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:44:31.112 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-03-24T11:44:31.169 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:44:31.169 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:31.169 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-24T11:44:31.169 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:31.169 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:31.170 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:44:23.259874011 +0000 2024-03-24T11:44:31.170 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:44:00.421321248 +0000 2024-03-24T11:44:31.170 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:44:00.421321248 +0000 2024-03-24T11:44:31.170 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:44:00.421321248 +0000 2024-03-24T11:44:31.170 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:44:31.237 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:44:31.237 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:44:31.238 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000229713 s, 2.2 MB/s 2024-03-24T11:44:31.239 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:44:31.295 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-03-24T11:44:31.351 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:44:31.351 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:31.351 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-24T11:44:31.351 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:31.351 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:31.351 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:44:23.259874011 +0000 2024-03-24T11:44:31.352 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:44:00.769314433 +0000 2024-03-24T11:44:31.352 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:44:00.769314433 +0000 2024-03-24T11:44:31.352 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:44:00.769314433 +0000 2024-03-24T11:44:31.352 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:44:31.418 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:44:31.418 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:44:31.418 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000194526 s, 2.6 MB/s 2024-03-24T11:44:31.419 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:44:31.476 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-03-24T11:44:31.530 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:44:31.531 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:44:31.531 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T11:44:31.531 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:44:31.531 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:44:31.531 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:44:23.259874011 +0000 2024-03-24T11:44:31.531 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:44:01.103307893 +0000 2024-03-24T11:44:31.531 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:44:01.103307893 +0000 2024-03-24T11:44:31.531 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:44:01.103307893 +0000 2024-03-24T11:44:31.531 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:44:31.596 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:44:31.596 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:44:31.596 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000199872 s, 2.6 MB/s 2024-03-24T11:44:31.597 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:44:31.653 DEBUG:teuthology.orchestra.run.smithi189:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-24T11:44:31.791 INFO:teuthology.orchestra.run.smithi189.stdout:loop 2024-03-24T11:44:31.793 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_1... 2024-03-24T11:44:31.793 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-24T11:44:31.830 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-24T11:44:31.868 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_11 2024-03-24T11:44:31.904 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_2... 2024-03-24T11:44:31.905 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-24T11:44:31.984 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-24T11:44:32.023 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_21 2024-03-24T11:44:32.064 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_3... 2024-03-24T11:44:32.065 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-24T11:44:32.144 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-24T11:44:32.183 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_31 2024-03-24T11:44:32.212 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_4... 2024-03-24T11:44:32.212 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-24T11:44:32.293 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-24T11:44:32.333 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_41 2024-03-24T11:44:32.370 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:44:32.370 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:44:32.424 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme list 2024-03-24T11:44:32.491 INFO:teuthology.orchestra.run.smithi189.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-24T11:44:32.491 INFO:teuthology.orchestra.run.smithi189.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-24T11:44:32.491 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme0n1 CVFT6232004L400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-24T11:44:32.491 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme1n1 37c0160f32037c6666cb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:32.491 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme2n1 6a933c568da872d56090 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:32.491 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme3n1 f93d6156db02768fcf9a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:32.491 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme4n1 60f92fa714c10f261997 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:44:32.492 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:44:32.492 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:44:32.492 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/scratch_devs 2024-03-24T11:44:32.559 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T11:44:32.654 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T11:44:32.654 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:44:32.654 INFO:tasks.cephadm:Cluster fsid is e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:44:32.654 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T11:44:32.655 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.130', 'mon.b': '172.21.15.174', 'mon.c': '172.21.15.189'} 2024-03-24T11:44:32.655 INFO:tasks.cephadm:First mon is mon.a on smithi130 2024-03-24T11:44:32.655 INFO:tasks.cephadm:First mgr is y 2024-03-24T11:44:32.655 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T11:44:32.655 DEBUG:teuthology.orchestra.run.smithi130:> sudo hostname $(hostname -s) 2024-03-24T11:44:32.688 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname $(hostname -s) 2024-03-24T11:44:32.721 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-03-24T11:44:32.748 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T11:44:32.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:44:32.975 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77704/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1667-gdb0330b1', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1667.gdb0330b1'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-16 11:55:36.797747', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-24T11:44:33.113 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 db0330b1e4e2470d52b750e251e55a522b4f7d69 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:44:33.114 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm 2024-03-24T11:44:33.115 DEBUG:teuthology.orchestra.run.smithi130:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:44:33.869 INFO:teuthology.orchestra.run.smithi130.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 11:44 /home/ubuntu/cephtest/cephadm 2024-03-24T11:44:33.870 DEBUG:teuthology.orchestra.run.smithi174:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:44:34.654 INFO:teuthology.orchestra.run.smithi174.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 11:44 /home/ubuntu/cephtest/cephadm 2024-03-24T11:44:34.655 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:44:35.443 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 11:44 /home/ubuntu/cephtest/cephadm 2024-03-24T11:44:35.444 DEBUG:teuthology.orchestra.run.smithi130:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:44:35.463 DEBUG:teuthology.orchestra.run.smithi174:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:44:35.481 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:44:35.512 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 on all hosts... 2024-03-24T11:44:35.512 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:44:35.515 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:44:35.524 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:44:36.357 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:44:36.469 INFO:teuthology.orchestra.run.smithi174.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:44:36.633 INFO:teuthology.orchestra.run.smithi130.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:45:10.004 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-03-24T11:45:10.004 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:45:10.005 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:45:10.005 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-03-24T11:45:10.005 INFO:teuthology.orchestra.run.smithi189.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:45:10.005 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-03-24T11:45:10.005 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-03-24T11:45:36.443 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-03-24T11:45:36.443 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:45:36.443 INFO:teuthology.orchestra.run.smithi130.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:45:36.443 INFO:teuthology.orchestra.run.smithi130.stdout: "repo_digests": [ 2024-03-24T11:45:36.443 INFO:teuthology.orchestra.run.smithi130.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:45:36.443 INFO:teuthology.orchestra.run.smithi130.stdout: ] 2024-03-24T11:45:36.443 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-03-24T11:45:37.015 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-03-24T11:45:37.015 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:45:37.015 INFO:teuthology.orchestra.run.smithi174.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:45:37.015 INFO:teuthology.orchestra.run.smithi174.stdout: "repo_digests": [ 2024-03-24T11:45:37.015 INFO:teuthology.orchestra.run.smithi174.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:45:37.015 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-03-24T11:45:37.015 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-03-24T11:45:37.039 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph 2024-03-24T11:45:37.072 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph 2024-03-24T11:45:37.101 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-03-24T11:45:37.137 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 777 /etc/ceph 2024-03-24T11:45:37.165 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /etc/ceph 2024-03-24T11:45:37.192 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-03-24T11:45:37.222 INFO:tasks.cephadm:Writing seed config... 2024-03-24T11:45:37.223 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T11:45:37.223 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T11:45:37.223 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-24T11:45:37.223 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-24T11:45:37.223 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T11:45:37.224 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T11:45:37.224 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T11:45:37.224 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T11:45:37.224 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T11:45:37.224 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T11:45:37.224 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:45:37.224 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T11:45:37.240 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e18cf2c8-e9d3-11ee-95cd-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-24T11:45:37.240 DEBUG:teuthology.orchestra.run.smithi130:mon.a> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T11:45:37.283 DEBUG:teuthology.orchestra.run.smithi130:mgr.y> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T11:45:37.326 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T11:45:37.326 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 -v bootstrap --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.130 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:45:37.518 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:45:37.519 INFO:teuthology.orchestra.run.smithi130.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69', '-v', 'bootstrap', '--fsid', 'e18cf2c8-e9d3-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.130', '--skip-admin-label'] 2024-03-24T11:45:37.543 INFO:teuthology.orchestra.run.smithi130.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:45:37.543 INFO:teuthology.orchestra.run.smithi130.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T11:45:37.544 INFO:teuthology.orchestra.run.smithi130.stdout:Verifying podman|docker is present... 2024-03-24T11:45:37.567 INFO:teuthology.orchestra.run.smithi130.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:45:37.568 INFO:teuthology.orchestra.run.smithi130.stdout:Verifying lvm2 is present... 2024-03-24T11:45:37.568 INFO:teuthology.orchestra.run.smithi130.stdout:Verifying time synchronization is in place... 2024-03-24T11:45:37.575 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:45:37.575 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:45:37.580 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:45:37.580 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout inactive 2024-03-24T11:45:37.586 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout enabled 2024-03-24T11:45:37.592 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout active 2024-03-24T11:45:37.593 INFO:teuthology.orchestra.run.smithi130.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:45:37.593 INFO:teuthology.orchestra.run.smithi130.stdout:Repeating the final host check... 2024-03-24T11:45:37.615 INFO:teuthology.orchestra.run.smithi130.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:45:37.615 INFO:teuthology.orchestra.run.smithi130.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T11:45:37.615 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl is present 2024-03-24T11:45:37.616 INFO:teuthology.orchestra.run.smithi130.stdout:lvcreate is present 2024-03-24T11:45:37.621 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:45:37.621 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:45:37.627 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:45:37.627 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout inactive 2024-03-24T11:45:37.633 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout enabled 2024-03-24T11:45:37.639 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout active 2024-03-24T11:45:37.640 INFO:teuthology.orchestra.run.smithi130.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:45:37.640 INFO:teuthology.orchestra.run.smithi130.stdout:Host looks OK 2024-03-24T11:45:37.640 INFO:teuthology.orchestra.run.smithi130.stdout:Cluster fsid: e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:45:37.640 INFO:teuthology.orchestra.run.smithi130.stdout:Acquiring lock 139628956329008 on /run/cephadm/e18cf2c8-e9d3-11ee-95cd-87774f69a715.lock 2024-03-24T11:45:37.640 INFO:teuthology.orchestra.run.smithi130.stdout:Lock 139628956329008 acquired on /run/cephadm/e18cf2c8-e9d3-11ee-95cd-87774f69a715.lock 2024-03-24T11:45:37.640 INFO:teuthology.orchestra.run.smithi130.stdout:Verifying IP 172.21.15.130 port 3300 ... 2024-03-24T11:45:37.640 INFO:teuthology.orchestra.run.smithi130.stdout:Verifying IP 172.21.15.130 port 6789 ... 2024-03-24T11:45:37.640 INFO:teuthology.orchestra.run.smithi130.stdout:Base mon IP(s) is [172.21.15.130:3300, 172.21.15.130:6789], mon addrv is [v2:172.21.15.130:3300,v1:172.21.15.130:6789] 2024-03-24T11:45:37.643 INFO:teuthology.orchestra.run.smithi130.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.130 metric 100 2024-03-24T11:45:37.643 INFO:teuthology.orchestra.run.smithi130.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.130 metric 100 2024-03-24T11:45:37.646 INFO:teuthology.orchestra.run.smithi130.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T11:45:37.646 INFO:teuthology.orchestra.run.smithi130.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-24T11:45:37.646 INFO:teuthology.orchestra.run.smithi130.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-03-24T11:45:37.648 INFO:teuthology.orchestra.run.smithi130.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T11:45:37.648 INFO:teuthology.orchestra.run.smithi130.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T11:45:37.648 INFO:teuthology.orchestra.run.smithi130.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:45:37.648 INFO:teuthology.orchestra.run.smithi130.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-24T11:45:37.648 INFO:teuthology.orchestra.run.smithi130.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7f37/64 scope link 2024-03-24T11:45:37.648 INFO:teuthology.orchestra.run.smithi130.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:45:37.649 INFO:teuthology.orchestra.run.smithi130.stdout:Mon IP `172.21.15.130` is in CIDR network `172.21.0.0/20` 2024-03-24T11:45:37.649 INFO:teuthology.orchestra.run.smithi130.stdout:Mon IP `172.21.15.130` is in CIDR network `172.21.0.0/20` 2024-03-24T11:45:37.649 INFO:teuthology.orchestra.run.smithi130.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T11:45:37.649 INFO:teuthology.orchestra.run.smithi130.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T11:45:37.650 INFO:teuthology.orchestra.run.smithi130.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:45:38.145 INFO:teuthology.orchestra.run.smithi130.stdout:/bin/podman: stdout 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:45:38.146 INFO:teuthology.orchestra.run.smithi130.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:45:38.146 INFO:teuthology.orchestra.run.smithi130.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T11:45:38.146 INFO:teuthology.orchestra.run.smithi130.stdout:/bin/podman: stderr Copying blob sha256:989de1c398e87dbcb4906791a352373dc0e0415d619bc5b36f1f1823df929cb1 2024-03-24T11:45:38.146 INFO:teuthology.orchestra.run.smithi130.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-24T11:45:38.146 INFO:teuthology.orchestra.run.smithi130.stdout:/bin/podman: stderr Copying config sha256:4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:45:38.146 INFO:teuthology.orchestra.run.smithi130.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T11:45:39.024 INFO:teuthology.orchestra.run.smithi130.stdout:ceph: stdout ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:45:39.024 INFO:teuthology.orchestra.run.smithi130.stdout:Ceph version: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:45:39.024 INFO:teuthology.orchestra.run.smithi130.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T11:45:40.054 INFO:teuthology.orchestra.run.smithi130.stdout:stat: stdout 167 167 2024-03-24T11:45:40.055 INFO:teuthology.orchestra.run.smithi130.stdout:Creating initial keys... 2024-03-24T11:45:40.925 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph-authtool: stdout AQBkEgBmPzjEFBAAXzFam7SeDn7WZfaGVERk3g== 2024-03-24T11:45:45.421 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph-authtool: stdout AQBlEgBmN7t+HBAAHhD+B3X6WDJpPdpYBLf5bw== 2024-03-24T11:45:49.961 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph-authtool: stdout AQBsEgBmyfEIGhAAVo5WFDubVzT1j82ubfSELw== 2024-03-24T11:45:49.962 INFO:teuthology.orchestra.run.smithi130.stdout:Creating initial monmap... 2024-03-24T11:45:50.842 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:45:50.842 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T11:45:50.842 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:45:50.842 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:45:50.842 INFO:teuthology.orchestra.run.smithi130.stdout:monmaptool for a [v2:172.21.15.130:3300,v1:172.21.15.130:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:45:50.846 INFO:teuthology.orchestra.run.smithi130.stdout:setting min_mon_release = quincy 2024-03-24T11:45:50.846 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/monmaptool: set fsid to e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:45:50.846 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:45:50.846 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:45:50.846 INFO:teuthology.orchestra.run.smithi130.stdout:Creating mon... 2024-03-24T11:45:51.812 INFO:teuthology.orchestra.run.smithi130.stdout:create mon.a on 2024-03-24T11:45:52.165 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T11:45:52.348 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715.target. 2024-03-24T11:45:52.349 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715.target. 2024-03-24T11:45:52.585 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.a 2024-03-24T11:45:52.585 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T11:45:52.743 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715.target.wants/ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@.service. 2024-03-24T11:45:53.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:52 smithi130 podman[25269]: 2024-03-24 11:45:52.887536478 +0000 UTC m=+0.017642050 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:45:53.368 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:53 smithi130 podman[25269]: 2024-03-24 11:45:53.24561301 +0000 UTC m=+0.375718563 container create ffce2a5ce478eb1728798ad0b00357841f5de7737a6bbbd63649cb74510a66c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T11:45:53.569 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:45:53.569 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout disabled 2024-03-24T11:45:53.576 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:45:53.576 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout inactive 2024-03-24T11:45:53.576 INFO:teuthology.orchestra.run.smithi130.stdout:firewalld.service is not enabled 2024-03-24T11:45:53.576 INFO:teuthology.orchestra.run.smithi130.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:45:53.576 INFO:teuthology.orchestra.run.smithi130.stdout:Waiting for mon to start... 2024-03-24T11:45:53.577 INFO:teuthology.orchestra.run.smithi130.stdout:Waiting for mon... 2024-03-24T11:45:53.702 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:53 smithi130 podman[25269]: 2024-03-24 11:45:53.447751905 +0000 UTC m=+0.577857466 container init ffce2a5ce478eb1728798ad0b00357841f5de7737a6bbbd63649cb74510a66c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T11:45:53.702 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:53 smithi130 podman[25269]: 2024-03-24 11:45:53.45334584 +0000 UTC m=+0.583451394 container start ffce2a5ce478eb1728798ad0b00357841f5de7737a6bbbd63649cb74510a66c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T11:45:53.702 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:53 smithi130 ceph-mon[25307]: mkfs e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:45:53.702 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:53 smithi130 ceph-mon[25307]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:45:53.702 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:53 smithi130 bash[25269]: ffce2a5ce478eb1728798ad0b00357841f5de7737a6bbbd63649cb74510a66c4 2024-03-24T11:45:53.702 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:53 smithi130 systemd[1]: Started Ceph mon.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:45:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:54 smithi130 ceph-mon[25307]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:45:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:54 smithi130 ceph-mon[25307]: monmap e1: 1 mons at {a=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:45:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:54 smithi130 ceph-mon[25307]: fsmap 2024-03-24T11:45:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:54 smithi130 ceph-mon[25307]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:45:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:54 smithi130 ceph-mon[25307]: mgrmap e1: no daemons active 2024-03-24T11:45:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:54 smithi130 ceph-mon[25307]: from='client.? 172.21.15.130:0/2110856701' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout id: e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout services: 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.563804s) 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout data: 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T11:45:55.791 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T11:45:55.792 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T11:45:55.792 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T11:45:55.792 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:45:55.792 INFO:teuthology.orchestra.run.smithi130.stdout:mon is available 2024-03-24T11:45:55.792 INFO:teuthology.orchestra.run.smithi130.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T11:45:56.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:56 smithi130 ceph-mon[25307]: from='client.? 172.21.15.130:0/128760934' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:45:56.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:56 smithi130 ceph-mon[25307]: from='client.? 172.21.15.130:0/128760934' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T11:45:56.828 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:45:56.828 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:45:56.828 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout fsid = e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:45:56.828 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.130:3300,v1:172.21.15.130:6789] 2024-03-24T11:45:56.828 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:45:56.828 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:45:56.828 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:45:56.829 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:45:56.829 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:45:56.829 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:45:56.829 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-03-24T11:45:56.829 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:45:56.829 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:45:56.829 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:45:56.829 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:45:56.829 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:45:56.829 INFO:teuthology.orchestra.run.smithi130.stdout:Generating new minimal ceph.conf... 2024-03-24T11:45:57.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:57 smithi130 ceph-mon[25307]: from='client.? 172.21.15.130:0/65600860' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:45:57.882 INFO:teuthology.orchestra.run.smithi130.stdout:Restarting the monitor... 2024-03-24T11:45:58.194 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:57 smithi130 systemd[1]: Stopping Ceph mon.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:45:58.195 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:58 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a[25284]: 2024-03-24T11:45:58.142+0000 7f2317d0c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:45:58.195 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:58 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a[25284]: 2024-03-24T11:45:58.142+0000 7f2317d0c700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T11:45:58.514 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:58 smithi130 podman[25639]: 2024-03-24 11:45:58.194596482 +0000 UTC m=+0.151068199 container died ffce2a5ce478eb1728798ad0b00357841f5de7737a6bbbd63649cb74510a66c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b) 2024-03-24T11:45:58.514 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:58 smithi130 podman[25639]: 2024-03-24 11:45:58.514584532 +0000 UTC m=+0.471056259 container cleanup ffce2a5ce478eb1728798ad0b00357841f5de7737a6bbbd63649cb74510a66c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T11:45:58.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:58 smithi130 bash[25639]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a 2024-03-24T11:45:58.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:58 smithi130 podman[25657]: 2024-03-24 11:45:58.749111897 +0000 UTC m=+0.556347619 container remove ffce2a5ce478eb1728798ad0b00357841f5de7737a6bbbd63649cb74510a66c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:45:59.178 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:58 smithi130 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T11:45:59.178 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:58 smithi130 systemd[1]: Stopped Ceph mon.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:45:59.178 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:58 smithi130 systemd[1]: Starting Ceph mon.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:45:59.178 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 podman[25757]: 2024-03-24 11:45:59.066641916 +0000 UTC m=+0.098529890 container create 360e2bed32ee5dbce3791bad8e60bef7b16b0d536660e8bdaf6785e4f5e4c7c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:45:59.178 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 podman[25757]: 2024-03-24 11:45:58.985812788 +0000 UTC m=+0.017700737 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:45:59.364 INFO:teuthology.orchestra.run.smithi130.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T11:45:59.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 podman[25757]: 2024-03-24 11:45:59.277192822 +0000 UTC m=+0.309080776 container init 360e2bed32ee5dbce3791bad8e60bef7b16b0d536660e8bdaf6785e4f5e4c7c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T11:45:59.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 podman[25757]: 2024-03-24 11:45:59.282134632 +0000 UTC m=+0.314022563 container start 360e2bed32ee5dbce3791bad8e60bef7b16b0d536660e8bdaf6785e4f5e4c7c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T11:45:59.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:45:59.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T11:45:59.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: pidfile_write: ignore empty --pid-file 2024-03-24T11:45:59.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: load: jerasure load: lrc 2024-03-24T11:45:59.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:45:59.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Git sha 0 2024-03-24T11:45:59.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:45:59.540 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: DB SUMMARY 2024-03-24T11:45:59.540 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: DB Session ID: C15ZF230TF5AISE7ZRHS 2024-03-24T11:45:59.540 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:45:59.540 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:45:59.540 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T11:45:59.540 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T11:45:59.540 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84641 ; 2024-03-24T11:45:59.540 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:45:59.540 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:45:59.540 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:45:59.541 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:45:59.541 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:45:59.541 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:45:59.541 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.env: 0x55e8d64edae0 2024-03-24T11:45:59.541 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:45:59.541 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.info_log: 0x55e8d83598c0 2024-03-24T11:45:59.541 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:45:59.541 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.statistics: (nil) 2024-03-24T11:45:59.543 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.use_fsync: 0 2024-03-24T11:45:59.543 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:45:59.543 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:45:59.543 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:45:59.543 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:45:59.543 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:45:59.543 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:45:59.543 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:45:59.543 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:45:59.543 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:45:59.544 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:45:59.544 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:45:59.544 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.db_log_dir: 2024-03-24T11:45:59.544 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.wal_dir: 2024-03-24T11:45:59.544 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:45:59.544 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:45:59.544 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:45:59.544 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:45:59.544 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:45:59.544 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:45:59.544 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.write_buffer_manager: 0x55e8d832d400 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.unordered_write: 0 2024-03-24T11:45:59.545 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:45:59.546 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:45:59.546 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:45:59.546 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:45:59.546 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.row_cache: None 2024-03-24T11:45:59.546 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.wal_filter: None 2024-03-24T11:45:59.546 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:45:59.546 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:45:59.546 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:45:59.546 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:45:59.546 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.wal_compression: 0 2024-03-24T11:45:59.546 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:45:59.547 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_open_files: -1 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:45:59.548 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Compression algorithms supported: 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: kZSTD supported: 0 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:45:59.549 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.merge_operator: 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_filter: None 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e8d83599c0) 2024-03-24T11:45:59.550 INFO:journalctl@ceph.mon.a.smithi130.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: index_type: 0 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: data_block_index_type: 0 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: index_shortening: 1 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: checksum: 4 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: no_block_cache: 0 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: block_cache: 0x55e8d835d090 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:45:59.551 INFO:journalctl@ceph.mon.a.smithi130.stdout: block_cache_options: 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: capacity : 536870912 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: num_shard_bits : 4 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: strict_capacity_limit : 0 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: block_cache_compressed: (nil) 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: persistent_cache: (nil) 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: block_size: 4096 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: block_size_deviation: 10 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: block_restart_interval: 16 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: index_block_restart_interval: 1 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: metadata_block_size: 4096 2024-03-24T11:45:59.552 INFO:journalctl@ceph.mon.a.smithi130.stdout: partition_filters: 0 2024-03-24T11:45:59.553 INFO:journalctl@ceph.mon.a.smithi130.stdout: use_delta_encoding: 1 2024-03-24T11:45:59.553 INFO:journalctl@ceph.mon.a.smithi130.stdout: filter_policy: bloomfilter 2024-03-24T11:45:59.553 INFO:journalctl@ceph.mon.a.smithi130.stdout: whole_key_filtering: 1 2024-03-24T11:45:59.553 INFO:journalctl@ceph.mon.a.smithi130.stdout: verify_compression: 0 2024-03-24T11:45:59.553 INFO:journalctl@ceph.mon.a.smithi130.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:45:59.553 INFO:journalctl@ceph.mon.a.smithi130.stdout: format_version: 5 2024-03-24T11:45:59.553 INFO:journalctl@ceph.mon.a.smithi130.stdout: enable_index_compression: 1 2024-03-24T11:45:59.553 INFO:journalctl@ceph.mon.a.smithi130.stdout: block_align: 0 2024-03-24T11:45:59.553 INFO:journalctl@ceph.mon.a.smithi130.stdout: max_auto_readahead_size: 262144 2024-03-24T11:45:59.553 INFO:journalctl@ceph.mon.a.smithi130.stdout: prepopulate_block_cache: 0 2024-03-24T11:45:59.553 INFO:journalctl@ceph.mon.a.smithi130.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compression: NoCompression 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.num_levels: 7 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:45:59.554 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:45:59.555 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:45:59.555 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:45:59.555 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:45:59.555 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:45:59.555 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:45:59.555 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:45:59.555 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:45:59.555 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:45:59.555 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:45:59.555 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:45:59.555 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:45:59.556 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:45:59.557 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:45:59.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:45:59.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:45:59.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:45:59.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:45:59.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:45:59.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:45:59.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:45:59.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:45:59.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:45:59.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:45:59.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:45:59.559 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.ttl: 2592000 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:45:59.560 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 73b9171c-de69-421a-8900-1ee5052345c8 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280759311657, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280759312497, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81266, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79453, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73762, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280759, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "73b9171c-de69-421a-8900-1ee5052345c8", "db_session_id": "C15ZF230TF5AISE7ZRHS", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280759312555, "job": 1, "event": "recovery_finished"} 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e8d844e000 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: DB pointer 0x55e8d8438000 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:45:59.561 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:45:59.562 INFO:journalctl@ceph.mon.a.smithi130.stdout: ** DB Stats ** 2024-03-24T11:45:59.562 INFO:journalctl@ceph.mon.a.smithi130.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:45:59.562 INFO:journalctl@ceph.mon.a.smithi130.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:45:59.562 INFO:journalctl@ceph.mon.a.smithi130.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:45:59.562 INFO:journalctl@ceph.mon.a.smithi130.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:45:59.562 INFO:journalctl@ceph.mon.a.smithi130.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:45:59.562 INFO:journalctl@ceph.mon.a.smithi130.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:45:59.562 INFO:journalctl@ceph.mon.a.smithi130.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:45:59.562 INFO:journalctl@ceph.mon.a.smithi130.stdout: 2024-03-24T11:45:59.562 INFO:journalctl@ceph.mon.a.smithi130.stdout: ** Compaction Stats [default] ** 2024-03-24T11:45:59.562 INFO:journalctl@ceph.mon.a.smithi130.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: L0 2/0 81.20 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: Sum 2/0 81.20 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: ** Compaction Stats [default] ** 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:45:59.563 INFO:journalctl@ceph.mon.a.smithi130.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:45:59.564 INFO:journalctl@ceph.mon.a.smithi130.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:45:59.564 INFO:journalctl@ceph.mon.a.smithi130.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:45:59.564 INFO:journalctl@ceph.mon.a.smithi130.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:45:59.564 INFO:journalctl@ceph.mon.a.smithi130.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:45:59.564 INFO:journalctl@ceph.mon.a.smithi130.stdout: Cumulative compaction: 0.00 GB write, 14.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:45:59.564 INFO:journalctl@ceph.mon.a.smithi130.stdout: Interval compaction: 0.00 GB write, 14.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:45:59.564 INFO:journalctl@ceph.mon.a.smithi130.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:45:59.564 INFO:journalctl@ceph.mon.a.smithi130.stdout: Block cache BinnedLRUCache@0x55e8d835d090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-24T11:45:59.564 INFO:journalctl@ceph.mon.a.smithi130.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:45:59.564 INFO:journalctl@ceph.mon.a.smithi130.stdout: 2024-03-24T11:45:59.564 INFO:journalctl@ceph.mon.a.smithi130.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: starting mon.a rank 0 at public addrs [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] at bind addrs [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: mon.a@-1(???) e1 preinit fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: mon.a@-1(???).mds e1 new map 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: mon.a@-1(???).mds e1 print_map 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout: e1 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout: legacy client fscid: -1 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout: 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout: No filesystems configured 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:45:59.565 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:45:59.566 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:45:59.566 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T11:45:59.566 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 bash[25757]: 360e2bed32ee5dbce3791bad8e60bef7b16b0d536660e8bdaf6785e4f5e4c7c2 2024-03-24T11:45:59.566 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 systemd[1]: Started Ceph mon.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:45:59.566 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:45:59.566 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: monmap e1: 1 mons at {a=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:45:59.566 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: fsmap 2024-03-24T11:45:59.566 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:45:59.566 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:45:59 smithi130 ceph-mon[25795]: mgrmap e1: no daemons active 2024-03-24T11:46:00.477 INFO:teuthology.orchestra.run.smithi130.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T11:46:00.499 INFO:teuthology.orchestra.run.smithi130.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:46:00.499 INFO:teuthology.orchestra.run.smithi130.stdout:Creating mgr... 2024-03-24T11:46:00.499 INFO:teuthology.orchestra.run.smithi130.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T11:46:00.500 INFO:teuthology.orchestra.run.smithi130.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T11:46:00.757 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.y 2024-03-24T11:46:00.757 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-24T11:46:00.908 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715.target.wants/ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@.service. 2024-03-24T11:46:01.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:00 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/4208636427' entity='client.admin' 2024-03-24T11:46:01.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:00 smithi130 systemd[1]: Starting Ceph mgr.y for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:46:01.536 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:01 smithi130 podman[26037]: 2024-03-24 11:46:01.04591515 +0000 UTC m=+0.018737113 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:46:01.537 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:01 smithi130 podman[26037]: 2024-03-24 11:46:01.284485153 +0000 UTC m=+0.257307110 container create 44df44b989736a7c6332e02d896794ab6ecb01238b9aad8d76fecc6c54ab51ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:46:01.750 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:46:01.750 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout disabled 2024-03-24T11:46:01.757 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:46:01.757 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout inactive 2024-03-24T11:46:01.757 INFO:teuthology.orchestra.run.smithi130.stdout:firewalld.service is not enabled 2024-03-24T11:46:01.757 INFO:teuthology.orchestra.run.smithi130.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:46:01.766 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:46:01.766 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout disabled 2024-03-24T11:46:01.773 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:46:01.773 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout inactive 2024-03-24T11:46:01.773 INFO:teuthology.orchestra.run.smithi130.stdout:firewalld.service is not enabled 2024-03-24T11:46:01.773 INFO:teuthology.orchestra.run.smithi130.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T11:46:01.773 INFO:teuthology.orchestra.run.smithi130.stdout:Waiting for mgr to start... 2024-03-24T11:46:01.773 INFO:teuthology.orchestra.run.smithi130.stdout:Waiting for mgr... 2024-03-24T11:46:01.904 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:01 smithi130 podman[26037]: 2024-03-24 11:46:01.653347544 +0000 UTC m=+0.626169502 container init 44df44b989736a7c6332e02d896794ab6ecb01238b9aad8d76fecc6c54ab51ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T11:46:01.904 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:01 smithi130 podman[26037]: 2024-03-24 11:46:01.659297593 +0000 UTC m=+0.632119551 container start 44df44b989736a7c6332e02d896794ab6ecb01238b9aad8d76fecc6c54ab51ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T11:46:01.904 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:01 smithi130 bash[26037]: 44df44b989736a7c6332e02d896794ab6ecb01238b9aad8d76fecc6c54ab51ec 2024-03-24T11:46:01.904 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:01 smithi130 systemd[1]: Started Ceph mgr.y for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:46:02.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:02 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/985834287' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:46:02.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:02 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:02.228+0000 7f925598d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:46:02.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:02 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:02.324+0000 7f925598d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:46:02.864 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:02.864 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:02.864 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "fsid": "e18cf2c8-e9d3-11ee-95cd-87774f69a715", 2024-03-24T11:46:02.864 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:46:02.864 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:46:02.864 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:46:02.864 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:46:02.864 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:02.864 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:46:02.864 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:46:02.864 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:46:02.865 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:46:02.869 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:46:02.869 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:46:02.869 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:46:02.869 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:02.869 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:46:02.869 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:46:02.869 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:46:02.869 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:46:02.870 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:45:53.525629+0000", 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:02.871 INFO:teuthology.orchestra.run.smithi130.stdout:mgr not available, waiting (1/15)... 2024-03-24T11:46:03.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:02 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:02.978+0000 7f925598d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:46:03.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:03 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:03.069+0000 7f925598d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:46:03.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:03 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:03.210+0000 7f925598d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:46:03.646 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:03 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:03.305+0000 7f925598d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:46:03.646 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:03 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:03.386+0000 7f925598d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:46:03.647 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:03 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:03.645+0000 7f925598d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:46:03.933 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:03 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:03.754+0000 7f925598d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:46:03.933 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:03 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:03.839+0000 7f925598d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:46:03.933 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:03 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:03.932+0000 7f925598d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:46:04.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:04 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:04.108+0000 7f925598d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:46:05.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:05 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:05.080+0000 7f925598d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:46:05.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:05 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:05.213+0000 7f925598d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:46:05.594 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:05 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/342150118' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:46:05.595 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:05 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:05.560+0000 7f925598d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:46:05.856 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:05 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:05.660+0000 7f925598d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:46:05.856 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:05 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:05.855+0000 7f925598d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:46:05.952 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:05.952 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:05.952 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "fsid": "e18cf2c8-e9d3-11ee-95cd-87774f69a715", 2024-03-24T11:46:05.952 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:46:05.952 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:46:05.952 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:46:05.952 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:46:05.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:05.954 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:46:05.954 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:05.954 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:46:05.954 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:46:05.954 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:46:05.954 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:46:05.954 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:46:05.954 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:46:05.958 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:05.958 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:46:05.958 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:46:05.958 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:46:05.958 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:46:05.958 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:46:05.958 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:46:05.958 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:46:05.958 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:46:05.958 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:46:05.958 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:05.959 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:05.960 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:46:05.960 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:05.960 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:45:53.525629+0000", 2024-03-24T11:46:05.960 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:05.960 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:05.960 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:46:05.960 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:05.960 INFO:teuthology.orchestra.run.smithi130.stdout:mgr not available, waiting (2/15)... 2024-03-24T11:46:06.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:06 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:06.041+0000 7f925598d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:46:06.585 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:06 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:06.323+0000 7f925598d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:46:06.585 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:06 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:06.584+0000 7f925598d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:46:07.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:06 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:06.817+0000 7f925598d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:46:08.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:07 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:07.610+0000 7f925598d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:46:08.341 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:08.187+0000 7f925598d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:46:08.341 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:08.291+0000 7f925598d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:46:08.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:08.508+0000 7f925598d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:46:08.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/516374078' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:46:08.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: Activating manager daemon y 2024-03-24T11:46:08.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: mgrmap e2: y(active, starting, since 0.00415776s) 2024-03-24T11:46:08.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: from='mgr.14100 172.21.15.130:0/728610544' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:46:08.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: from='mgr.14100 172.21.15.130:0/728610544' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:46:08.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: from='mgr.14100 172.21.15.130:0/728610544' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:46:08.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: from='mgr.14100 172.21.15.130:0/728610544' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:46:08.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: from='mgr.14100 172.21.15.130:0/728610544' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:46:08.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: Manager daemon y is now available 2024-03-24T11:46:08.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: from='mgr.14100 172.21.15.130:0/728610544' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:46:08.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: from='mgr.14100 172.21.15.130:0/728610544' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:46:08.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: from='mgr.14100 172.21.15.130:0/728610544' entity='mgr.y' 2024-03-24T11:46:08.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: from='mgr.14100 172.21.15.130:0/728610544' entity='mgr.y' 2024-03-24T11:46:08.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:08 smithi130 ceph-mon[25795]: from='mgr.14100 172.21.15.130:0/728610544' entity='mgr.y' 2024-03-24T11:46:09.074 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "fsid": "e18cf2c8-e9d3-11ee-95cd-87774f69a715", 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:46:09.075 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:46:09.076 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:46:09.080 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:09.081 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:09.082 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:46:09.082 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:09.082 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:45:53.525629+0000", 2024-03-24T11:46:09.082 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:09.082 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:09.082 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:46:09.082 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:09.082 INFO:teuthology.orchestra.run.smithi130.stdout:mgr not available, waiting (3/15)... 2024-03-24T11:46:10.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:10 smithi130 ceph-mon[25795]: mgrmap e3: y(active, since 1.0075s) 2024-03-24T11:46:12.607 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:12.607 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:12.607 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "fsid": "e18cf2c8-e9d3-11ee-95cd-87774f69a715", 2024-03-24T11:46:12.607 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:46:12.607 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:46:12.607 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:46:12.607 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:46:12.607 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:12.607 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:46:12.607 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:46:12.608 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:46:12.609 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:46:12.609 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:46:12.612 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:45:53.525629+0000", 2024-03-24T11:46:12.613 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:46:12.614 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:46:12.614 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:46:12.614 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:12.614 INFO:teuthology.orchestra.run.smithi130.stdout:mgr is available 2024-03-24T11:46:12.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:12 smithi130 ceph-mon[25795]: mgrmap e4: y(active, since 3s) 2024-03-24T11:46:12.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:12 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/1877612856' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:46:13.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:13 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/4280513414' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:46:13.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:13 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/4280513414' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T11:46:13.835 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout fsid = e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:46:13.836 INFO:teuthology.orchestra.run.smithi130.stdout:Enabling cephadm module... 2024-03-24T11:46:14.888 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:14 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/2373420997' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T11:46:14.888 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:14 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: ignoring --setuser ceph since I am not root 2024-03-24T11:46:14.888 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:14 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: ignoring --setgroup ceph since I am not root 2024-03-24T11:46:15.399 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:15 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:15.147+0000 7f8eff5a3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:46:15.400 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:15 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:15.246+0000 7f8eff5a3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:46:15.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:15 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/2373420997' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T11:46:15.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:15 smithi130 ceph-mon[25795]: mgrmap e5: y(active, since 6s) 2024-03-24T11:46:16.210 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:15 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:15.904+0000 7f8eff5a3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:46:16.211 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:16 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:16.002+0000 7f8eff5a3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:46:16.211 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:16 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:16.146+0000 7f8eff5a3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:46:16.493 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:16 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:16.243+0000 7f8eff5a3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:46:16.493 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:16 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:16.325+0000 7f8eff5a3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:46:16.496 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:16.496 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T11:46:16.496 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:46:16.497 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T11:46:16.497 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:46:16.497 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:16.497 INFO:teuthology.orchestra.run.smithi130.stdout:Waiting for the mgr to restart... 2024-03-24T11:46:16.497 INFO:teuthology.orchestra.run.smithi130.stdout:Waiting for mgr epoch 5... 2024-03-24T11:46:16.751 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:16 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3239146629' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:46:16.751 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:16 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:16.594+0000 7f8eff5a3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:46:16.751 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:16 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:16.707+0000 7f8eff5a3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:46:17.078 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:16 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:16.802+0000 7f8eff5a3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:46:17.079 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:16 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:16.898+0000 7f8eff5a3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:46:17.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:17.078+0000 7f8eff5a3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:46:18.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:18 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:18.031+0000 7f8eff5a3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:46:18.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:18 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:18.160+0000 7f8eff5a3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:46:18.777 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:18 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:18.494+0000 7f8eff5a3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:46:18.777 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:18 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:18.583+0000 7f8eff5a3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:46:18.777 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:18 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:18.776+0000 7f8eff5a3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:46:19.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:18 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:18.959+0000 7f8eff5a3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:46:19.500 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:19 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:19.240+0000 7f8eff5a3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:46:19.501 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:19 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:19.499+0000 7f8eff5a3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:46:19.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:19 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:19.732+0000 7f8eff5a3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:46:20.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:20 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:20.520+0000 7f8eff5a3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:46:21.398 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:21.087+0000 7f8eff5a3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:46:21.399 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:21.184+0000 7f8eff5a3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:46:21.399 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:21.397+0000 7f8eff5a3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:46:21.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: Active manager daemon y restarted 2024-03-24T11:46:21.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: Activating manager daemon y 2024-03-24T11:46:21.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T11:46:21.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: mgrmap e6: y(active, starting, since 0.00391508s) 2024-03-24T11:46:21.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:46:21.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:46:21.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:46:21.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:46:21.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:46:21.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: Manager daemon y is now available 2024-03-24T11:46:21.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:21.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:21.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:21.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:21 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:22.724 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:22 smithi130 ceph-mon[25795]: Found migration_current of "None". Setting to last migration. 2024-03-24T11:46:22.724 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:22 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:22.724 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:22 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:46:22.724 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:22 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:46:22.724 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:22 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:22.724 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:22 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:22.724 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:22 smithi130 ceph-mon[25795]: mgrmap e7: y(active, since 1.0052s) 2024-03-24T11:46:22.952 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:22.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T11:46:22.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:46:22.953 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:22.953 INFO:teuthology.orchestra.run.smithi130.stdout:mgr epoch 5 is available 2024-03-24T11:46:22.953 INFO:teuthology.orchestra.run.smithi130.stdout:Setting orchestrator backend to cephadm... 2024-03-24T11:46:23.673 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:23 smithi130 ceph-mon[25795]: [24/Mar/2024:11:46:22] ENGINE Bus STARTING 2024-03-24T11:46:23.673 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:23 smithi130 ceph-mon[25795]: [24/Mar/2024:11:46:22] ENGINE Serving on https://172.21.15.130:7150 2024-03-24T11:46:23.673 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:23 smithi130 ceph-mon[25795]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:46:23.673 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:23 smithi130 ceph-mon[25795]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:46:23.673 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:23 smithi130 ceph-mon[25795]: [24/Mar/2024:11:46:22] ENGINE Serving on http://172.21.15.130:8765 2024-03-24T11:46:23.673 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:23 smithi130 ceph-mon[25795]: [24/Mar/2024:11:46:22] ENGINE Bus STARTED 2024-03-24T11:46:23.673 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:23 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:24.563 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:24 smithi130 ceph-mon[25795]: mgrmap e8: y(active, since 2s) 2024-03-24T11:46:24.563 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:24 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:24.564 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:24 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:25.429 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T11:46:25.430 INFO:teuthology.orchestra.run.smithi130.stdout:Generating ssh key... 2024-03-24T11:46:25.787 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:25 smithi130 ceph-mon[25795]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:26.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-mon[25795]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:26.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: Generating public/private rsa key pair. 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: Your identification has been saved in /tmp/tmpmxlcu2r9/key. 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: Your public key has been saved in /tmp/tmpmxlcu2r9/key.pub. 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: The key fingerprint is: 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: SHA256:8y4UQ7bi2UXK/S9HNANN+5LNPFfFOm22OpRhSvTLR2E ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: The key's randomart image is: 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: +---[RSA 3072]----+ 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: | o...| 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: | o ... .Eo| 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: | + =. ..ooo| 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: | . * o. +*B=| 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: | . +S+..+.BBB| 2024-03-24T11:46:26.789 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: | o oo ..=.+o| 2024-03-24T11:46:26.790 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: | . . .oo | 2024-03-24T11:46:26.790 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: | .. .oo | 2024-03-24T11:46:26.790 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: | .. o. | 2024-03-24T11:46:26.790 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: +----[SHA256]-----+ 2024-03-24T11:46:27.668 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:27 smithi130 ceph-mon[25795]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:27.668 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:27 smithi130 ceph-mon[25795]: Generating ssh key... 2024-03-24T11:46:27.668 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:27 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:27.669 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:27 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:28.457 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQGh7tvt8kMt+xdOjZul/WPvYjzvvHDy3MStFccEW2sLwf52Jn+030HH+T7kmGJ43BNrGQv1/44eSfmZNRY0UUe8pkd3RdFkrWJ8EoSrp9neQBeWje98wJEGtzHPiPYocRwba565Xlxm/fhZkJrE+zwpRY2VwRyV9N+eA2/xL17MQnD36RTN3EgJ8dd2K69ZoKK79pmFL76OAzncQFZmfqkJMGPm9FqT3FcMFT19xGqJep30jP4ODJmblj5fS9paAUjvafWeYNt7j0Qj34nPLdHpI2P54NafdtfGdluIf/WTX6Z30RAQqCzdEE0BYjzz3VxEHCuEDWi5Ue1T1w5+cZ3QoK/xx1v9xD6z9XePLNKI0q84xv1IP0ZhhTYqMvVp5mYBrUfa/1iMk47NTrkENl50b1V2iT+KknDbh/YkuWEn214KS6JBEFyZjc6R1+WBWSGgJoE+yILPFTsPwKZrO0mr26l6Bdmr91xOgDghAvZzATo7J/SUubZmbYByPXa50= ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:46:28.458 INFO:teuthology.orchestra.run.smithi130.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:46:28.458 INFO:teuthology.orchestra.run.smithi130.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T11:46:28.458 INFO:teuthology.orchestra.run.smithi130.stdout:Adding host smithi130... 2024-03-24T11:46:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:28 smithi130 ceph-mon[25795]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:29.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:29 smithi130 ceph-mon[25795]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "addr": "172.21.15.130", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:31.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:30 smithi130 ceph-mon[25795]: Deploying cephadm binary to smithi130 2024-03-24T11:46:33.939 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout Added host 'smithi130' with addr '172.21.15.130' 2024-03-24T11:46:33.939 INFO:teuthology.orchestra.run.smithi130.stdout:Deploying unmanaged mon service... 2024-03-24T11:46:34.635 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:34 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:34.635 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:34 smithi130 ceph-mon[25795]: Added host smithi130 2024-03-24T11:46:34.635 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:34 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:35.272 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T11:46:35.272 INFO:teuthology.orchestra.run.smithi130.stdout:Deploying unmanaged mgr service... 2024-03-24T11:46:35.873 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:35 smithi130 ceph-mon[25795]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:35.873 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:35 smithi130 ceph-mon[25795]: Saving service mon spec with placement count:5 2024-03-24T11:46:35.873 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:35 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:36.579 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T11:46:37.147 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:36 smithi130 ceph-mon[25795]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:37.147 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:36 smithi130 ceph-mon[25795]: Saving service mgr spec with placement count:2 2024-03-24T11:46:37.148 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:36 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:38.536 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:38 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/2215622911' entity='client.admin' 2024-03-24T11:46:39.450 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:39 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:39.450 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:39 smithi130 ceph-mon[25795]: from='mgr.14120 172.21.15.130:0/921476032' entity='mgr.y' 2024-03-24T11:46:39.450 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:39 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/1624268031' entity='client.admin' 2024-03-24T11:46:39.703 INFO:teuthology.orchestra.run.smithi130.stdout:Enabling the dashboard module... 2024-03-24T11:46:40.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:40 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/4053268678' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T11:46:40.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:40 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: ignoring --setuser ceph since I am not root 2024-03-24T11:46:40.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:40 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: ignoring --setgroup ceph since I am not root 2024-03-24T11:46:41.116 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:41 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:41.094+0000 7fe6983f3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:46:41.458 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:41 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:41.192+0000 7fe6983f3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:46:41.723 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:41 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/4053268678' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T11:46:41.724 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:41 smithi130 ceph-mon[25795]: mgrmap e9: y(active, since 19s) 2024-03-24T11:46:42.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:41 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:41.850+0000 7fe6983f3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:46:42.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:41 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:41.947+0000 7fe6983f3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:46:42.524 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:42.094+0000 7fe6983f3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:46:42.524 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:42.192+0000 7fe6983f3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:46:42.524 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:42.279+0000 7fe6983f3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:46:42.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:42 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3467483172' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:46:42.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:42.546+0000 7fe6983f3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:46:42.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:42.656+0000 7fe6983f3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:46:42.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:42.741+0000 7fe6983f3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:46:42.860 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:42.860 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T11:46:42.861 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:46:42.861 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T11:46:42.861 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:46:42.861 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:42.861 INFO:teuthology.orchestra.run.smithi130.stdout:Waiting for the mgr to restart... 2024-03-24T11:46:42.861 INFO:teuthology.orchestra.run.smithi130.stdout:Waiting for mgr epoch 9... 2024-03-24T11:46:43.098 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:42.837+0000 7fe6983f3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:46:43.098 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:43 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:43.022+0000 7fe6983f3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:46:44.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:43 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:43.989+0000 7fe6983f3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:46:44.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:44 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:44.119+0000 7fe6983f3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:46:44.738 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:44 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:44.452+0000 7fe6983f3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:46:44.739 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:44 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:44.542+0000 7fe6983f3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:46:44.739 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:44 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:44.737+0000 7fe6983f3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:46:45.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:44 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:44.922+0000 7fe6983f3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:46:45.469 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:45 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:45.207+0000 7fe6983f3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:46:45.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:45 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:45.467+0000 7fe6983f3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:46:45.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:45 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:45.705+0000 7fe6983f3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:46:46.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:46 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:46.497+0000 7fe6983f3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:46:47.377 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:47.061+0000 7fe6983f3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:46:47.377 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:47.160+0000 7fe6983f3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:46:47.378 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:46:47.375+0000 7fe6983f3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:46:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: Active manager daemon y restarted 2024-03-24T11:46:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: Activating manager daemon y 2024-03-24T11:46:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T11:46:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: mgrmap e10: y(active, starting, since 0.00336598s) 2024-03-24T11:46:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:46:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:46:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:46:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:46:47.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:46:47.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: Manager daemon y is now available 2024-03-24T11:46:47.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:47.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:46:48.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:48 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:46:48.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:48 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:46:48.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:48 smithi130 ceph-mon[25795]: mgrmap e11: y(active, since 1.00553s) 2024-03-24T11:46:49.042 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:46:49.042 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T11:46:49.042 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:46:49.042 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:46:49.042 INFO:teuthology.orchestra.run.smithi130.stdout:mgr epoch 9 is available 2024-03-24T11:46:49.042 INFO:teuthology.orchestra.run.smithi130.stdout:Generating a dashboard self-signed certificate... 2024-03-24T11:46:49.743 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:49 smithi130 ceph-mon[25795]: [24/Mar/2024:11:46:48] ENGINE Bus STARTING 2024-03-24T11:46:49.743 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:49 smithi130 ceph-mon[25795]: [24/Mar/2024:11:46:48] ENGINE Serving on http://172.21.15.130:8765 2024-03-24T11:46:49.743 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:49 smithi130 ceph-mon[25795]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:46:49.743 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:49 smithi130 ceph-mon[25795]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:46:49.743 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:49 smithi130 ceph-mon[25795]: [24/Mar/2024:11:46:48] ENGINE Serving on https://172.21.15.130:7150 2024-03-24T11:46:49.743 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:49 smithi130 ceph-mon[25795]: [24/Mar/2024:11:46:48] ENGINE Bus STARTED 2024-03-24T11:46:49.743 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:46:50.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:50 smithi130 ceph-mon[25795]: mgrmap e12: y(active, since 2s) 2024-03-24T11:46:50.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:50 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:46:50.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:50 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:46:50.806 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T11:46:50.807 INFO:teuthology.orchestra.run.smithi130.stdout:Creating initial admin user... 2024-03-24T11:46:51.719 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:51 smithi130 ceph-mon[25795]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:51.719 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:51 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:46:52.287 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$fkP.cyhLh.bvt4CZ5cu5O.RmayfO0U3CrMEOf1z40bEt448z6B4V2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711280811, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T11:46:52.287 INFO:teuthology.orchestra.run.smithi130.stdout:Fetching dashboard port number... 2024-03-24T11:46:52.537 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:52 smithi130 ceph-mon[25795]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:46:52.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:52 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:46:53.611 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:53 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/225781555' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T11:46:53.854 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T11:46:53.864 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:46:53.908 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout disabled 2024-03-24T11:46:53.910 INFO:teuthology.orchestra.run.smithi130.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:46:53.910 INFO:teuthology.orchestra.run.smithi130.stdout:systemctl: stdout inactive 2024-03-24T11:46:53.910 INFO:teuthology.orchestra.run.smithi130.stdout:firewalld.service is not enabled 2024-03-24T11:46:53.910 INFO:teuthology.orchestra.run.smithi130.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T11:46:53.910 INFO:teuthology.orchestra.run.smithi130.stdout:Ceph Dashboard is now available at: 2024-03-24T11:46:53.910 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:46:53.910 INFO:teuthology.orchestra.run.smithi130.stdout: URL: https://smithi130.front.sepia.ceph.com:8443/ 2024-03-24T11:46:53.910 INFO:teuthology.orchestra.run.smithi130.stdout: User: admin 2024-03-24T11:46:53.910 INFO:teuthology.orchestra.run.smithi130.stdout: Password: 2f7iqiq6i1 2024-03-24T11:46:53.911 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:46:53.911 INFO:teuthology.orchestra.run.smithi130.stdout:Saving cluster configuration to /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config directory 2024-03-24T11:46:53.911 INFO:teuthology.orchestra.run.smithi130.stdout:Enabling autotune for osd_memory_target 2024-03-24T11:46:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:54 smithi130 ceph-mon[25795]: mgrmap e13: y(active, since 6s) 2024-03-24T11:46:56.777 INFO:teuthology.orchestra.run.smithi130.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T11:46:56.777 INFO:teuthology.orchestra.run.smithi130.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T11:46:56.777 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:46:56.777 INFO:teuthology.orchestra.run.smithi130.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:46:56.777 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:46:56.777 INFO:teuthology.orchestra.run.smithi130.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T11:46:56.777 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:46:56.778 INFO:teuthology.orchestra.run.smithi130.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T11:46:56.778 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:46:56.778 INFO:teuthology.orchestra.run.smithi130.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T11:46:56.778 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:46:56.778 INFO:teuthology.orchestra.run.smithi130.stdout: ceph telemetry on 2024-03-24T11:46:56.778 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:46:56.778 INFO:teuthology.orchestra.run.smithi130.stdout:For more information see: 2024-03-24T11:46:56.778 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:46:56.778 INFO:teuthology.orchestra.run.smithi130.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T11:46:56.778 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:46:56.778 INFO:teuthology.orchestra.run.smithi130.stdout:Bootstrap complete. 2024-03-24T11:46:56.819 INFO:tasks.cephadm:Fetching config... 2024-03-24T11:46:56.820 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:46:56.820 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T11:46:56.834 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T11:46:56.835 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:46:56.835 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T11:46:56.889 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T11:46:56.889 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:46:56.889 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T11:46:56.958 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T11:46:56.958 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:46:56.958 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T11:46:57.012 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T11:46:57.012 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQGh7tvt8kMt+xdOjZul/WPvYjzvvHDy3MStFccEW2sLwf52Jn+030HH+T7kmGJ43BNrGQv1/44eSfmZNRY0UUe8pkd3RdFkrWJ8EoSrp9neQBeWje98wJEGtzHPiPYocRwba565Xlxm/fhZkJrE+zwpRY2VwRyV9N+eA2/xL17MQnD36RTN3EgJ8dd2K69ZoKK79pmFL76OAzncQFZmfqkJMGPm9FqT3FcMFT19xGqJep30jP4ODJmblj5fS9paAUjvafWeYNt7j0Qj34nPLdHpI2P54NafdtfGdluIf/WTX6Z30RAQqCzdEE0BYjzz3VxEHCuEDWi5Ue1T1w5+cZ3QoK/xx1v9xD6z9XePLNKI0q84xv1IP0ZhhTYqMvVp5mYBrUfa/1iMk47NTrkENl50b1V2iT+KknDbh/YkuWEn214KS6JBEFyZjc6R1+WBWSGgJoE+yILPFTsPwKZrO0mr26l6Bdmr91xOgDghAvZzATo7J/SUubZmbYByPXa50= ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:46:57.095 INFO:teuthology.orchestra.run.smithi130.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQGh7tvt8kMt+xdOjZul/WPvYjzvvHDy3MStFccEW2sLwf52Jn+030HH+T7kmGJ43BNrGQv1/44eSfmZNRY0UUe8pkd3RdFkrWJ8EoSrp9neQBeWje98wJEGtzHPiPYocRwba565Xlxm/fhZkJrE+zwpRY2VwRyV9N+eA2/xL17MQnD36RTN3EgJ8dd2K69ZoKK79pmFL76OAzncQFZmfqkJMGPm9FqT3FcMFT19xGqJep30jP4ODJmblj5fS9paAUjvafWeYNt7j0Qj34nPLdHpI2P54NafdtfGdluIf/WTX6Z30RAQqCzdEE0BYjzz3VxEHCuEDWi5Ue1T1w5+cZ3QoK/xx1v9xD6z9XePLNKI0q84xv1IP0ZhhTYqMvVp5mYBrUfa/1iMk47NTrkENl50b1V2iT+KknDbh/YkuWEn214KS6JBEFyZjc6R1+WBWSGgJoE+yILPFTsPwKZrO0mr26l6Bdmr91xOgDghAvZzATo7J/SUubZmbYByPXa50= ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:46:57.124 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQGh7tvt8kMt+xdOjZul/WPvYjzvvHDy3MStFccEW2sLwf52Jn+030HH+T7kmGJ43BNrGQv1/44eSfmZNRY0UUe8pkd3RdFkrWJ8EoSrp9neQBeWje98wJEGtzHPiPYocRwba565Xlxm/fhZkJrE+zwpRY2VwRyV9N+eA2/xL17MQnD36RTN3EgJ8dd2K69ZoKK79pmFL76OAzncQFZmfqkJMGPm9FqT3FcMFT19xGqJep30jP4ODJmblj5fS9paAUjvafWeYNt7j0Qj34nPLdHpI2P54NafdtfGdluIf/WTX6Z30RAQqCzdEE0BYjzz3VxEHCuEDWi5Ue1T1w5+cZ3QoK/xx1v9xD6z9XePLNKI0q84xv1IP0ZhhTYqMvVp5mYBrUfa/1iMk47NTrkENl50b1V2iT+KknDbh/YkuWEn214KS6JBEFyZjc6R1+WBWSGgJoE+yILPFTsPwKZrO0mr26l6Bdmr91xOgDghAvZzATo7J/SUubZmbYByPXa50= ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:46:57.169 INFO:teuthology.orchestra.run.smithi174.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQGh7tvt8kMt+xdOjZul/WPvYjzvvHDy3MStFccEW2sLwf52Jn+030HH+T7kmGJ43BNrGQv1/44eSfmZNRY0UUe8pkd3RdFkrWJ8EoSrp9neQBeWje98wJEGtzHPiPYocRwba565Xlxm/fhZkJrE+zwpRY2VwRyV9N+eA2/xL17MQnD36RTN3EgJ8dd2K69ZoKK79pmFL76OAzncQFZmfqkJMGPm9FqT3FcMFT19xGqJep30jP4ODJmblj5fS9paAUjvafWeYNt7j0Qj34nPLdHpI2P54NafdtfGdluIf/WTX6Z30RAQqCzdEE0BYjzz3VxEHCuEDWi5Ue1T1w5+cZ3QoK/xx1v9xD6z9XePLNKI0q84xv1IP0ZhhTYqMvVp5mYBrUfa/1iMk47NTrkENl50b1V2iT+KknDbh/YkuWEn214KS6JBEFyZjc6R1+WBWSGgJoE+yILPFTsPwKZrO0mr26l6Bdmr91xOgDghAvZzATo7J/SUubZmbYByPXa50= ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:46:57.185 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQGh7tvt8kMt+xdOjZul/WPvYjzvvHDy3MStFccEW2sLwf52Jn+030HH+T7kmGJ43BNrGQv1/44eSfmZNRY0UUe8pkd3RdFkrWJ8EoSrp9neQBeWje98wJEGtzHPiPYocRwba565Xlxm/fhZkJrE+zwpRY2VwRyV9N+eA2/xL17MQnD36RTN3EgJ8dd2K69ZoKK79pmFL76OAzncQFZmfqkJMGPm9FqT3FcMFT19xGqJep30jP4ODJmblj5fS9paAUjvafWeYNt7j0Qj34nPLdHpI2P54NafdtfGdluIf/WTX6Z30RAQqCzdEE0BYjzz3VxEHCuEDWi5Ue1T1w5+cZ3QoK/xx1v9xD6z9XePLNKI0q84xv1IP0ZhhTYqMvVp5mYBrUfa/1iMk47NTrkENl50b1V2iT+KknDbh/YkuWEn214KS6JBEFyZjc6R1+WBWSGgJoE+yILPFTsPwKZrO0mr26l6Bdmr91xOgDghAvZzATo7J/SUubZmbYByPXa50= ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:46:57.229 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQGh7tvt8kMt+xdOjZul/WPvYjzvvHDy3MStFccEW2sLwf52Jn+030HH+T7kmGJ43BNrGQv1/44eSfmZNRY0UUe8pkd3RdFkrWJ8EoSrp9neQBeWje98wJEGtzHPiPYocRwba565Xlxm/fhZkJrE+zwpRY2VwRyV9N+eA2/xL17MQnD36RTN3EgJ8dd2K69ZoKK79pmFL76OAzncQFZmfqkJMGPm9FqT3FcMFT19xGqJep30jP4ODJmblj5fS9paAUjvafWeYNt7j0Qj34nPLdHpI2P54NafdtfGdluIf/WTX6Z30RAQqCzdEE0BYjzz3VxEHCuEDWi5Ue1T1w5+cZ3QoK/xx1v9xD6z9XePLNKI0q84xv1IP0ZhhTYqMvVp5mYBrUfa/1iMk47NTrkENl50b1V2iT+KknDbh/YkuWEn214KS6JBEFyZjc6R1+WBWSGgJoE+yILPFTsPwKZrO0mr26l6Bdmr91xOgDghAvZzATo7J/SUubZmbYByPXa50= ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:46:57.244 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T11:46:57.368 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:46:57 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/19735958' entity='client.admin' 2024-03-24T11:46:57.487 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:00.617 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:00 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:00.617 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:00 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:00.617 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:00 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:47:00.618 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:00 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:00.618 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:00 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:00.618 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:00 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:00.618 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:00 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3206723429' entity='client.admin' 2024-03-24T11:47:00.618 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:00 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:00.618 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:00 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:00.644 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T11:47:00.644 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T11:47:00.879 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:03.343 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi174 2024-03-24T11:47:03.344 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-24T11:47:03.344 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.conf 2024-03-24T11:47:03.362 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-24T11:47:03.362 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:03.419 INFO:tasks.cephadm:Adding host smithi174 to orchestrator... 2024-03-24T11:47:03.419 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch host add smithi174 2024-03-24T11:47:03.665 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:03.884 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:03.884 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:03.884 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:03.884 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:03.884 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:03.884 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: Updating smithi130:/etc/ceph/ceph.conf 2024-03-24T11:47:03.884 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: Updating smithi130:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:47:03.884 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:03.884 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: Updating smithi130:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:47:03.884 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:03.885 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:03.885 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:06.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:05 smithi130 ceph-mon[25795]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:06.926 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:06 smithi130 ceph-mon[25795]: Deploying cephadm binary to smithi174 2024-03-24T11:47:08.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:08 smithi130 ceph-mon[25795]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:09.729 INFO:teuthology.orchestra.run.smithi130.stdout:Added host 'smithi174' with addr '172.21.15.174' 2024-03-24T11:47:10.297 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T11:47:10.537 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:10.659 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:10 smithi130 ceph-mon[25795]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:10.659 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:10 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:10.659 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:10 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:10.659 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:10 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:11.630 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:11 smithi130 ceph-mon[25795]: Added host smithi174 2024-03-24T11:47:11.631 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:11 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:12.336 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:47:12.336 INFO:teuthology.orchestra.run.smithi130.stdout:[{"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}] 2024-03-24T11:47:12.678 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:12 smithi130 ceph-mon[25795]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:12.678 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:12 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:12.991 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-03-24T11:47:12.991 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:47:12.991 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-03-24T11:47:13.009 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:47:13.009 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:13.067 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-03-24T11:47:13.068 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch host add smithi189 2024-03-24T11:47:13.304 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:13.672 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:13 smithi130 ceph-mon[25795]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:47:14.691 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:14 smithi130 ceph-mon[25795]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:16.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:16.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:16.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:16.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:16.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:16.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:47:16.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:16.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:16.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-24T11:47:16.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:16.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:47:16.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:16.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:16.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:17.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:17 smithi130 ceph-mon[25795]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:17.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:17 smithi130 ceph-mon[25795]: Deploying cephadm binary to smithi189 2024-03-24T11:47:17.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:17 smithi130 ceph-mon[25795]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:47:18.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:18 smithi130 ceph-mon[25795]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:19.397 INFO:teuthology.orchestra.run.smithi130.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-03-24T11:47:19.973 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T11:47:20.206 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:20.657 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:20.658 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:20 smithi130 ceph-mon[25795]: Added host smithi189 2024-03-24T11:47:20.658 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:20 smithi130 ceph-mon[25795]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:20.658 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:20.658 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:20.658 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:21.990 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:47:21.990 INFO:teuthology.orchestra.run.smithi130.stdout:[{"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-03-24T11:47:22.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:22 smithi130 ceph-mon[25795]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:22.558 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:22 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:22.584 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T11:47:22.585 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T11:47:22.815 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:23.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:23 smithi130 ceph-mon[25795]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:47:24.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:24 smithi130 ceph-mon[25795]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:25.488 INFO:teuthology.orchestra.run.smithi130.stderr:adjusted tunables profile to default 2024-03-24T11:47:25.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:25 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/1473643478' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:47:25.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:25.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:25.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:25.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:25.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:47:25.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:25.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:26.080 INFO:tasks.cephadm:Adding mon.a on smithi130 2024-03-24T11:47:26.080 INFO:tasks.cephadm:Adding mon.b on smithi174 2024-03-24T11:47:26.080 INFO:tasks.cephadm:Adding mon.c on smithi189 2024-03-24T11:47:26.080 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi130:172.21.15.130=a;smithi174:172.21.15.174=b;smithi189:172.21.15.189=c' 2024-03-24T11:47:26.309 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:47:26.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:26 smithi130 ceph-mon[25795]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:47:26.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:26 smithi130 ceph-mon[25795]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:47:26.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:26 smithi130 ceph-mon[25795]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:26.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:26 smithi130 ceph-mon[25795]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:26.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:26 smithi130 ceph-mon[25795]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:47:26.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:26 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/1473643478' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:47:26.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:26 smithi130 ceph-mon[25795]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:26.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:26 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:26.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:26 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:26.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:26 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:27.892 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mon update... 2024-03-24T11:47:28.447 DEBUG:teuthology.orchestra.run.smithi174:mon.b> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T11:47:28.449 DEBUG:teuthology.orchestra.run.smithi189:mon.c> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T11:47:28.452 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:47:28.452 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:47:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:28 smithi130 ceph-mon[25795]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:28 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:28 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:28 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:28 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:28 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:28 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:28 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:29.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:29 smithi130 ceph-mon[25795]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi130:172.21.15.130=a;smithi174:172.21.15.174=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:29.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:29 smithi130 ceph-mon[25795]: Saving service mon spec with placement smithi130:172.21.15.130=a;smithi174:172.21.15.174=b;smithi189:172.21.15.189=c;count:3 2024-03-24T11:47:29.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:29 smithi130 ceph-mon[25795]: Deploying daemon mon.c on smithi189 2024-03-24T11:47:30.554 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:47:30.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:30 smithi130 ceph-mon[25795]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:32.335 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 podman[30561]: 2024-03-24 11:47:32.222591584 +0000 UTC m=+0.312594444 container create 9df70d4d8c048f4be99decb4ba739fd6293d7769dc20ff38b9f1c61ebbc134d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-c, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T11:47:32.668 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 podman[30561]: 2024-03-24 11:47:32.500474766 +0000 UTC m=+0.590477627 container init 9df70d4d8c048f4be99decb4ba739fd6293d7769dc20ff38b9f1c61ebbc134d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-c, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:47:32.668 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 podman[30561]: 2024-03-24 11:47:32.505857985 +0000 UTC m=+0.595860847 container start 9df70d4d8c048f4be99decb4ba739fd6293d7769dc20ff38b9f1c61ebbc134d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_BRANCH=HEAD) 2024-03-24T11:47:32.668 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:47:32.668 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T11:47:32.668 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: pidfile_write: ignore empty --pid-file 2024-03-24T11:47:32.668 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: load: jerasure load: lrc 2024-03-24T11:47:32.668 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Git sha 0 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: DB SUMMARY 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: DB Session ID: 2JQSQGD0HFP0VV4PEGQ8 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:47:32.669 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:47:32.670 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.env: 0x55e4c9f0dae0 2024-03-24T11:47:32.670 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:47:32.670 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.info_log: 0x55e4cc0798c0 2024-03-24T11:47:32.670 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.statistics: (nil) 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.use_fsync: 0 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:47:32.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.db_log_dir: 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.wal_dir: 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.write_buffer_manager: 0x55e4cc04d400 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:47:32.672 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.unordered_write: 0 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.row_cache: None 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.wal_filter: None 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:47:32.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.wal_compression: 0 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:47:32.674 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_open_files: -1 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:47:32.675 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Compression algorithms supported: 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: kZSTD supported: 0 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:47:32.676 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.merge_operator: 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_filter: None 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e4cc0799c0) 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_type: 0 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout: data_block_index_type: 0 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_shortening: 1 2024-03-24T11:47:32.677 INFO:journalctl@ceph.mon.c.smithi189.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: checksum: 4 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: no_block_cache: 0 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache: 0x55e4cc07d090 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_options: 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: capacity : 536870912 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: num_shard_bits : 4 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: strict_capacity_limit : 0 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_compressed: (nil) 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: persistent_cache: (nil) 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_size: 4096 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_size_deviation: 10 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_restart_interval: 16 2024-03-24T11:47:32.678 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_block_restart_interval: 1 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: metadata_block_size: 4096 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: partition_filters: 0 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: use_delta_encoding: 1 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: filter_policy: bloomfilter 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: whole_key_filtering: 1 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: verify_compression: 0 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: format_version: 5 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: enable_index_compression: 1 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_align: 0 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: max_auto_readahead_size: 262144 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: prepopulate_block_cache: 0 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:47:32.679 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compression: NoCompression 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.num_levels: 7 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:47:32.680 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:47:32.681 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:47:32.682 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:47:32.682 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:47:32.682 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:47:32.682 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:47:32.682 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:47:32.682 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:47:32.683 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:47:32.683 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:47:32.683 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:47:32.683 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:47:32.683 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:47:32.683 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:47:32.683 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:47:32.683 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:47:32.684 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.ttl: 2592000 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:47:32.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ae69b96e-865d-4fd2-a2d3-ba54f6582d33 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280852536553, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T11:47:32.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280852536965, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280852, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ae69b96e-865d-4fd2-a2d3-ba54f6582d33", "db_session_id": "2JQSQGD0HFP0VV4PEGQ8", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280852537046, "job": 1, "event": "recovery_finished"} 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e4cc16e000 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: DB pointer 0x55e4cc158000 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** DB Stats ** 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:47:32.687 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** Compaction Stats [default] ** 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** Compaction Stats [default] ** 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:47:32.688 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: Block cache BinnedLRUCache@0x55e4cc07d090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:47:32.689 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: using public_addr v2:172.21.15.189:0/0 -> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: starting mon.c rank -1 at public addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] at bind addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(???) e0 preinit fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).mds e1 new map 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout: e1 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout: legacy client fscid: -1 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout: No filesystems configured 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T11:47:32.690 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:47:32.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Deploying cephadm binary to smithi189 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Added host smithi189 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:32.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/1473643478' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:47:32.693 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/1473643478' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:32.694 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:32.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:32.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:32.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi130:172.21.15.130=a;smithi174:172.21.15.174=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:32.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Saving service mon spec with placement smithi130:172.21.15.130=a;smithi174:172.21.15.174=b;smithi189:172.21.15.189=c;count:3 2024-03-24T11:47:32.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: Deploying daemon mon.c on smithi189 2024-03-24T11:47:32.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:32.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:32.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 ceph-mon[30629]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T11:47:32.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 bash[30561]: 9df70d4d8c048f4be99decb4ba739fd6293d7769dc20ff38b9f1c61ebbc134d6 2024-03-24T11:47:32.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:32 smithi189 systemd[1]: Started Ceph mon.c for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:47:32.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:32 smithi130 ceph-mon[25795]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:37.596 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:47:37.596 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"e18cf2c8-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:47:32.559574Z","created":"2024-03-24T11:45:50.311749Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:47:37.599 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-03-24T11:47:37.921 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:47:37.921 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: mon.a calling monitor election 2024-03-24T11:47:37.921 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:37.921 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='client.? 172.21.15.189:0/2287107794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:47:37.921 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:37.921 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:37.921 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:37.921 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: mon.c calling monitor election 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: monmap e2: 2 mons at {a=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: fsmap 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: mgrmap e13: y(active, since 50s) 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: overall HEALTH_OK 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:37.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:37.923 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:37 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:38.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:47:38.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: mon.a calling monitor election 2024-03-24T11:47:38.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:38.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='client.? 172.21.15.189:0/2287107794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:47:38.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:38.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: mon.c calling monitor election 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: monmap e2: 2 mons at {a=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: fsmap 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: mgrmap e13: y(active, since 50s) 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: overall HEALTH_OK 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:38.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:38.040 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:38.040 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:37 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:38.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:38 smithi189 ceph-mon[30629]: Deploying daemon mon.b on smithi174 2024-03-24T11:47:38.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:38 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:39.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:38 smithi130 ceph-mon[25795]: Deploying daemon mon.b on smithi174 2024-03-24T11:47:39.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:38 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:39.143 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:47:39.143 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:47:39.392 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:47:39.848 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:39 smithi189 ceph-mon[30629]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:40.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:39 smithi130 ceph-mon[25795]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.157 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:47:41.157 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"e18cf2c8-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:47:32.559574Z","created":"2024-03-24T11:45:50.311749Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:47:41.159 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-03-24T11:47:41.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:41 smithi189 ceph-mon[30629]: from='client.? 172.21.15.189:0/631561831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:47:41.451 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 podman[30139]: 2024-03-24 11:47:41.253343005 +0000 UTC m=+0.017339891 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:47:41.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:41 smithi130 ceph-mon[25795]: from='client.? 172.21.15.189:0/631561831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:47:41.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 podman[30139]: 2024-03-24 11:47:41.550332768 +0000 UTC m=+0.314329660 container init fa9693ddc64f06f827549c5ab3fbe214355ef8ee8c9314c457235cdfa818c2ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T11:47:41.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 podman[30139]: 2024-03-24 11:47:41.55621612 +0000 UTC m=+0.320213000 container start fa9693ddc64f06f827549c5ab3fbe214355ef8ee8c9314c457235cdfa818c2ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T11:47:41.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:47:41.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T11:47:41.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pidfile_write: ignore empty --pid-file 2024-03-24T11:47:41.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: load: jerasure load: lrc 2024-03-24T11:47:41.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:47:41.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Git sha 0 2024-03-24T11:47:41.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:47:41.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: DB SUMMARY 2024-03-24T11:47:41.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: DB Session ID: DXGRRBIX6BABJN2Q0H1L 2024-03-24T11:47:41.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:47:41.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:47:41.846 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T11:47:41.846 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-24T11:47:41.846 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-24T11:47:41.846 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:47:41.846 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:47:41.846 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:47:41.846 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:47:41.846 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:47:41.846 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:47:41.846 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.env: 0x55fbb070fae0 2024-03-24T11:47:41.846 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:47:41.847 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.info_log: 0x55fbb22d18c0 2024-03-24T11:47:41.847 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:47:41.848 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.statistics: (nil) 2024-03-24T11:47:41.848 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.use_fsync: 0 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.db_log_dir: 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.wal_dir: 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:47:41.849 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.write_buffer_manager: 0x55fbb22a5400 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:47:41.850 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.unordered_write: 0 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.row_cache: None 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.wal_filter: None 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.wal_compression: 0 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:47:41.851 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:47:41.852 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_open_files: -1 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Compression algorithms supported: 2024-03-24T11:47:41.853 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: kZSTD supported: 0 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:47:41.854 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.merge_operator: 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_filter: None 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fbb22d19c0) 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout: index_type: 0 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout: data_block_index_type: 0 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout: index_shortening: 1 2024-03-24T11:47:41.855 INFO:journalctl@ceph.mon.b.smithi174.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: checksum: 4 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: no_block_cache: 0 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_cache: 0x55fbb22d5090 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_cache_options: 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: capacity : 536870912 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: num_shard_bits : 4 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: strict_capacity_limit : 0 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_cache_compressed: (nil) 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: persistent_cache: (nil) 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_size: 4096 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_size_deviation: 10 2024-03-24T11:47:41.856 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_restart_interval: 16 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: index_block_restart_interval: 1 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: metadata_block_size: 4096 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: partition_filters: 0 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: use_delta_encoding: 1 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: filter_policy: bloomfilter 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: whole_key_filtering: 1 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: verify_compression: 0 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: format_version: 5 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: enable_index_compression: 1 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: block_align: 0 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: max_auto_readahead_size: 262144 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: prepopulate_block_cache: 0 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:47:41.857 INFO:journalctl@ceph.mon.b.smithi174.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compression: NoCompression 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.num_levels: 7 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:47:41.858 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:47:41.859 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:47:41.860 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:47:41.861 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.ttl: 2592000 2024-03-24T11:47:41.862 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:47:41.863 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 00b95678-1ac8-44a1-bd1c-2d979e1e9511 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280861597057, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280861597459, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711280861, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "00b95678-1ac8-44a1-bd1c-2d979e1e9511", "db_session_id": "DXGRRBIX6BABJN2Q0H1L", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711280861597538, "job": 1, "event": "recovery_finished"} 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fbb23c6000 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: DB pointer 0x55fbb23b0000 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout: ** DB Stats ** 2024-03-24T11:47:41.864 INFO:journalctl@ceph.mon.b.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: ** Compaction Stats [default] ** 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-24T11:47:41.865 INFO:journalctl@ceph.mon.b.smithi174.stdout: ** Compaction Stats [default] ** 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:47:41.866 INFO:journalctl@ceph.mon.b.smithi174.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout: Block cache BinnedLRUCache@0x55fbb22d5090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: using public_addr v2:172.21.15.174:0/0 -> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: starting mon.b rank -1 at public addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] at bind addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(???) e0 preinit fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).mds e1 new map 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout: e1 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:47:41.867 INFO:journalctl@ceph.mon.b.smithi174.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout: legacy client fscid: -1 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout: 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout: No filesystems configured 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:47:41.868 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.869 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Deploying cephadm binary to smithi189 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Added host smithi189 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.870 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/1473643478' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.871 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/1473643478' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi130:172.21.15.130=a;smithi174:172.21.15.174=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:41.872 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Saving service mon spec with placement smithi130:172.21.15.130=a;smithi174:172.21.15.174=b;smithi189:172.21.15.189=c;count:3 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Deploying daemon mon.c on smithi189 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.a calling monitor election 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='client.? 172.21.15.189:0/2287107794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.c calling monitor election 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:41.873 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: monmap e2: 2 mons at {a=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: fsmap 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mgrmap e13: y(active, since 50s) 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: overall HEALTH_OK 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: Deploying daemon mon.b on smithi174 2024-03-24T11:47:41.874 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:41.875 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:41.875 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: from='client.? 172.21.15.189:0/631561831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:47:41.875 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 ceph-mon[30177]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T11:47:41.875 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 bash[30139]: fa9693ddc64f06f827549c5ab3fbe214355ef8ee8c9314c457235cdfa818c2ab 2024-03-24T11:47:41.875 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:41 smithi174 systemd[1]: Started Ceph mon.b for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:47:42.751 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:47:42.751 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:47:42.993 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:47:46.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:47:46.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: mon.a calling monitor election 2024-03-24T11:47:46.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: mon.c calling monitor election 2024-03-24T11:47:46.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:46.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:46.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:46.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:46.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:46.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:46.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: monmap e3: 3 mons at {a=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: fsmap 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: mgrmap e13: y(active, since 59s) 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: mon.b (rank 2) addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] is down (out of quorum) 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:46.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:46.948 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:47.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:47:47.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: mon.a calling monitor election 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: mon.c calling monitor election 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: monmap e3: 3 mons at {a=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: fsmap 2024-03-24T11:47:47.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:47.040 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: mgrmap e13: y(active, since 59s) 2024-03-24T11:47:47.040 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T11:47:47.040 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T11:47:47.040 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:47:47.040 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: mon.b (rank 2) addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] is down (out of quorum) 2024-03-24T11:47:47.040 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:47.040 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:47.040 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:47.040 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:47.765 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:47:47.765 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":3,"fsid":"e18cf2c8-e9d3-11ee-95cd-87774f69a715","modified":"2024-03-24T11:47:41.620318Z","created":"2024-03-24T11:45:50.311749Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:47:47.767 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 3 2024-03-24T11:47:47.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:47 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:47.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:47 smithi189 ceph-mon[30629]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:47.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:47 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:47.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:47 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:48.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:48.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:47 smithi130 ceph-mon[25795]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:48.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:48.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:48.781 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T11:47:48.781 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T11:47:49.026 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:49.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:47:49.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: mon.a calling monitor election 2024-03-24T11:47:49.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: mon.c calling monitor election 2024-03-24T11:47:49.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:49.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:47:49.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:49.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: monmap e3: 3 mons at {a=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: fsmap 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: mgrmap e13: y(active, since 59s) 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:47:49.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: mon.b (rank 2) addr [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] is down (out of quorum) 2024-03-24T11:47:49.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:49.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:49.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:49.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: mon.b calling monitor election 2024-03-24T11:47:49.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: mon.b calling monitor election 2024-03-24T11:47:49.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: mon.c calling monitor election 2024-03-24T11:47:49.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: mon.a calling monitor election 2024-03-24T11:47:49.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:47:49.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: monmap e3: 3 mons at {a=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:47:49.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: fsmap 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: mgrmap e13: y(active, since 61s) 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: Cluster is now healthy 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: overall HEALTH_OK 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.448 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.448 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:49.448 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:47:49.448 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:49.471 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: mon.b calling monitor election 2024-03-24T11:47:49.471 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: mon.b calling monitor election 2024-03-24T11:47:49.471 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: mon.c calling monitor election 2024-03-24T11:47:49.471 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: mon.a calling monitor election 2024-03-24T11:47:49.471 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:47:49.471 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: monmap e3: 3 mons at {a=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:47:49.471 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: fsmap 2024-03-24T11:47:49.471 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: mgrmap e13: y(active, since 61s) 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: Cluster is now healthy 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: overall HEALTH_OK 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.472 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:49.473 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:47:49.473 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:49.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: mon.b calling monitor election 2024-03-24T11:47:49.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: mon.b calling monitor election 2024-03-24T11:47:49.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: mon.c calling monitor election 2024-03-24T11:47:49.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: mon.a calling monitor election 2024-03-24T11:47:49.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:47:49.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: monmap e3: 3 mons at {a=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],b=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:47:49.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: fsmap 2024-03-24T11:47:49.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: mgrmap e13: y(active, since 61s) 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: Cluster is now healthy 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: overall HEALTH_OK 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.596 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:49.596 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:49.596 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:47:49.596 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:50.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:50 smithi130 ceph-mon[25795]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:47:50.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:50 smithi130 ceph-mon[25795]: Reconfiguring daemon mon.a on smithi130 2024-03-24T11:47:50.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:50 smithi130 ceph-mon[25795]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:50.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:50 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:50.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:50 smithi174 ceph-mon[30177]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:47:50.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:50 smithi174 ceph-mon[30177]: Reconfiguring daemon mon.a on smithi130 2024-03-24T11:47:50.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:50 smithi174 ceph-mon[30177]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:50.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:50 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:50.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:50 smithi189 ceph-mon[30629]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:47:50.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:50 smithi189 ceph-mon[30629]: Reconfiguring daemon mon.a on smithi130 2024-03-24T11:47:50.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:50 smithi189 ceph-mon[30629]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:50.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:50 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:47:51.069 INFO:teuthology.orchestra.run.smithi130.stdout:# minimal ceph.conf for e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:47:51.069 INFO:teuthology.orchestra.run.smithi130.stdout:[global] 2024-03-24T11:47:51.069 INFO:teuthology.orchestra.run.smithi130.stdout: fsid = e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:47:51.070 INFO:teuthology.orchestra.run.smithi130.stdout: mon_host = [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-03-24T11:47:51.491 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:51 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/1473087569' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:51.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:51 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/1473087569' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:51.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:51 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/1473087569' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:52.174 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T11:47:52.174 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:47:52.174 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:47:52.204 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:47:52.204 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:52.272 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-24T11:47:52.272 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:47:52.304 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-24T11:47:52.304 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:52.371 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:47:52.371 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:47:52.405 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:47:52.405 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:47:52.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:52 smithi189 ceph-mon[30629]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:52.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:52 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:52.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:52 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:52.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:52 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:52.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:52 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:47:52.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:52 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:52.475 INFO:tasks.cephadm:Adding mgr.y on smithi130 2024-03-24T11:47:52.476 INFO:tasks.cephadm:Adding mgr.x on smithi174 2024-03-24T11:47:52.476 INFO:tasks.cephadm:Adding mgr.z on smithi189 2024-03-24T11:47:52.476 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi130=y;smithi174=x;smithi189=z' 2024-03-24T11:47:52.525 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:52 smithi174 ceph-mon[30177]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:52.525 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:52 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:52.525 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:52 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:52.525 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:52 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:52.525 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:52 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:47:52.525 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:52 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:52.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:52 smithi130 ceph-mon[25795]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:52.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:52 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:52.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:52 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:52.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:52 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:52.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:52 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:47:52.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:52 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:52.749 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:47:53.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:53 smithi189 ceph-mon[30629]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:47:53.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:53 smithi189 ceph-mon[30629]: Reconfiguring daemon mon.b on smithi174 2024-03-24T11:47:53.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:53 smithi130 ceph-mon[25795]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:47:53.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:53 smithi130 ceph-mon[25795]: Reconfiguring daemon mon.b on smithi174 2024-03-24T11:47:53.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:53 smithi174 ceph-mon[30177]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:47:53.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:53 smithi174 ceph-mon[30177]: Reconfiguring daemon mon.b on smithi174 2024-03-24T11:47:54.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:54 smithi189 ceph-mon[30629]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:54.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:54 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:54.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:54 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:54.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:54 smithi189 ceph-mon[30629]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:47:54.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:54 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:54.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:54 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:47:54.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:54 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:54.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:54 smithi189 ceph-mon[30629]: Reconfiguring daemon mon.c on smithi189 2024-03-24T11:47:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:54 smithi130 ceph-mon[25795]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:54 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:54 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:54 smithi130 ceph-mon[25795]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:47:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:54 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:54 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:47:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:54 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:54.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:54 smithi130 ceph-mon[25795]: Reconfiguring daemon mon.c on smithi189 2024-03-24T11:47:54.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:54 smithi174 ceph-mon[30177]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:54.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:54 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:54.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:54 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:54.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:54 smithi174 ceph-mon[30177]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:47:54.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:54 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:47:54.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:54 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:47:54.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:54 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:54.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:54 smithi174 ceph-mon[30177]: Reconfiguring daemon mon.c on smithi189 2024-03-24T11:47:55.193 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mgr update... 2024-03-24T11:47:55.949 DEBUG:teuthology.orchestra.run.smithi174:mgr.x> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T11:47:55.952 DEBUG:teuthology.orchestra.run.smithi189:mgr.z> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.z.service 2024-03-24T11:47:55.954 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T11:47:55.954 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:47:55.954 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:47:55.969 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:47:55.969 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/nvme1n1 2024-03-24T11:47:56.024 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/nvme1n1 2024-03-24T11:47:56.025 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:56.025 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-24T11:47:56.025 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:56.025 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:56.025 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-24 11:46:57.766701672 +0000 2024-03-24T11:47:56.025 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-24 11:44:28.683394716 +0000 2024-03-24T11:47:56.025 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-24 11:44:28.683394716 +0000 2024-03-24T11:47:56.025 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-24 11:44:28.672394913 +0000 2024-03-24T11:47:56.025 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-24T11:47:56.091 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-24T11:47:56.092 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-24T11:47:56.092 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000315894 s, 1.6 MB/s 2024-03-24T11:47:56.093 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-24T11:47:56.149 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/nvme2n1 2024-03-24T11:47:56.203 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/nvme2n1 2024-03-24T11:47:56.203 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:56.203 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-24T11:47:56.203 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:56.203 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:56.204 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-24 11:46:57.776701493 +0000 2024-03-24T11:47:56.204 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-24 11:44:28.820392267 +0000 2024-03-24T11:47:56.204 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-24 11:44:28.820392267 +0000 2024-03-24T11:47:56.204 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-24 11:44:28.815392356 +0000 2024-03-24T11:47:56.204 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-24T11:47:56.269 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-24T11:47:56.269 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-24T11:47:56.269 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000231125 s, 2.2 MB/s 2024-03-24T11:47:56.270 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-24T11:47:56.326 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/nvme3n1 2024-03-24T11:47:56.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi130=y;smithi174=x;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:56.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: Saving service mgr spec with placement smithi130=y;smithi174=x;smithi189=z;count:3 2024-03-24T11:47:56.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:56.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:56.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:56.345 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:56.345 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.345 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:47:56.345 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:47:56.345 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:47:56.345 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:56 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:56.380 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/nvme3n1 2024-03-24T11:47:56.380 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:56.380 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-24T11:47:56.380 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:56.381 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:56.381 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-24 11:46:57.788701279 +0000 2024-03-24T11:47:56.381 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-24 11:44:28.970389585 +0000 2024-03-24T11:47:56.381 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-24 11:44:28.970389585 +0000 2024-03-24T11:47:56.381 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-24 11:44:28.959389781 +0000 2024-03-24T11:47:56.381 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-24T11:47:56.444 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi130=y;smithi174=x;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:56.444 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: Saving service mgr spec with placement smithi130=y;smithi174=x;smithi189=z;count:3 2024-03-24T11:47:56.444 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.444 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.444 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.445 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:56.445 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:56.445 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:56.445 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:56.445 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.445 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:47:56.445 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:47:56.445 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:47:56.445 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:56 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:56.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi130=y;smithi174=x;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:47:56.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: Saving service mgr spec with placement smithi130=y;smithi174=x;smithi189=z;count:3 2024-03-24T11:47:56.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:56.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:47:56.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:56.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:47:56.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:56.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:47:56.449 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:47:56.449 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:47:56.449 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:56 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:56.450 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-24T11:47:56.450 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-24T11:47:56.450 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000205352 s, 2.5 MB/s 2024-03-24T11:47:56.450 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-24T11:47:56.506 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/nvme4n1 2024-03-24T11:47:56.561 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/nvme4n1 2024-03-24T11:47:56.561 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:56.561 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-24T11:47:56.561 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:56.561 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:56.561 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-24 11:46:57.798701101 +0000 2024-03-24T11:47:56.561 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-24 11:44:29.102387224 +0000 2024-03-24T11:47:56.561 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-24 11:44:29.102387224 +0000 2024-03-24T11:47:56.561 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-24 11:44:29.101387242 +0000 2024-03-24T11:47:56.561 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-24T11:47:56.625 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-24T11:47:56.626 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-24T11:47:56.626 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.00021096 s, 2.4 MB/s 2024-03-24T11:47:56.627 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-24T11:47:56.683 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-24T11:47:56.683 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:47:56.702 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:47:56.702 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/nvme1n1 2024-03-24T11:47:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/nvme1n1 2024-03-24T11:47:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:56.761 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-24T11:47:56.761 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:56.761 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:56.761 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-24 11:47:14.198343048 +0000 2024-03-24T11:47:56.761 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-24 11:44:30.293439800 +0000 2024-03-24T11:47:56.761 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-24 11:44:30.293439800 +0000 2024-03-24T11:47:56.761 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-24 11:44:30.281439472 +0000 2024-03-24T11:47:56.761 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-24T11:47:56.830 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-24T11:47:56.830 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-24T11:47:56.830 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000251535 s, 2.0 MB/s 2024-03-24T11:47:56.832 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-24T11:47:56.890 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/nvme2n1 2024-03-24T11:47:56.948 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/nvme2n1 2024-03-24T11:47:56.948 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:56.948 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-24T11:47:56.948 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:56.948 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:56.948 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-24 11:47:14.209342907 +0000 2024-03-24T11:47:56.948 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-24 11:44:30.458444313 +0000 2024-03-24T11:47:56.948 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-24 11:44:30.458444313 +0000 2024-03-24T11:47:56.949 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-24 11:44:30.447444012 +0000 2024-03-24T11:47:56.949 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-24T11:47:57.017 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-24T11:47:57.017 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-24T11:47:57.017 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000281073 s, 1.8 MB/s 2024-03-24T11:47:57.019 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-24T11:47:57.076 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/nvme3n1 2024-03-24T11:47:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/nvme3n1 2024-03-24T11:47:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:57.133 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-24T11:47:57.133 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:57.133 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:57.133 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-24 11:47:14.219342779 +0000 2024-03-24T11:47:57.133 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-24 11:44:30.576447541 +0000 2024-03-24T11:47:57.133 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-24 11:44:30.576447541 +0000 2024-03-24T11:47:57.133 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-24 11:44:30.565447240 +0000 2024-03-24T11:47:57.134 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-24T11:47:57.201 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-24T11:47:57.201 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-24T11:47:57.201 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000231648 s, 2.2 MB/s 2024-03-24T11:47:57.202 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-24T11:47:57.259 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/nvme4n1 2024-03-24T11:47:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/nvme4n1 2024-03-24T11:47:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:57.315 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-24T11:47:57.315 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:57.315 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:57.315 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-03-24 11:47:14.233342601 +0000 2024-03-24T11:47:57.315 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-03-24 11:44:30.725446687 +0000 2024-03-24T11:47:57.315 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-03-24 11:44:30.725446687 +0000 2024-03-24T11:47:57.315 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-03-24 11:44:30.714446829 +0000 2024-03-24T11:47:57.316 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-24T11:47:57.382 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-03-24T11:47:57.382 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-03-24T11:47:57.382 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000218139 s, 2.3 MB/s 2024-03-24T11:47:57.383 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-24T11:47:57.439 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:47:57.439 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:47:57.456 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:47:57.456 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme1n1 2024-03-24T11:47:57.511 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme1n1 2024-03-24T11:47:57.512 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:57.512 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-24T11:47:57.512 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:57.512 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:57.512 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:47:23.700264248 +0000 2024-03-24T11:47:57.512 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:44:31.912582401 +0000 2024-03-24T11:47:57.512 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:44:31.912582401 +0000 2024-03-24T11:47:57.512 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:44:31.902582595 +0000 2024-03-24T11:47:57.512 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-24T11:47:57.562 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:57 smithi189 ceph-mon[30629]: Deploying daemon mgr.z on smithi189 2024-03-24T11:47:57.579 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:47:57.579 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:47:57.579 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.00028419 s, 1.8 MB/s 2024-03-24T11:47:57.580 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-24T11:47:57.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:57 smithi174 ceph-mon[30177]: Deploying daemon mgr.z on smithi189 2024-03-24T11:47:57.637 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme2n1 2024-03-24T11:47:57.691 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme2n1 2024-03-24T11:47:57.691 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:57.691 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 776 Links: 1 Device type: 103,4 2024-03-24T11:47:57.691 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:57.692 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:57.692 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:47:23.712264017 +0000 2024-03-24T11:47:57.692 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:44:32.071579324 +0000 2024-03-24T11:47:57.692 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:44:32.071579324 +0000 2024-03-24T11:47:57.692 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:44:32.062579498 +0000 2024-03-24T11:47:57.692 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-24T11:47:57.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:57 smithi130 ceph-mon[25795]: Deploying daemon mgr.z on smithi189 2024-03-24T11:47:57.831 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:47:57.831 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:47:57.831 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000258554 s, 2.0 MB/s 2024-03-24T11:47:57.832 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-24T11:47:57.848 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme3n1 2024-03-24T11:47:57.907 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme3n1 2024-03-24T11:47:57.908 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:57.908 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-24T11:47:57.908 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:57.908 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:57.908 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:47:23.722263825 +0000 2024-03-24T11:47:57.908 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:44:32.222576401 +0000 2024-03-24T11:47:57.908 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:44:32.222576401 +0000 2024-03-24T11:47:57.908 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:44:32.210576633 +0000 2024-03-24T11:47:57.908 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-24T11:47:58.015 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:47:58.015 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:47:58.015 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000202573 s, 2.5 MB/s 2024-03-24T11:47:58.017 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-24T11:47:58.032 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme4n1 2024-03-24T11:47:58.088 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme4n1 2024-03-24T11:47:58.088 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:47:58.088 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-24T11:47:58.088 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:47:58.088 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:47:58.088 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:47:23.733263614 +0000 2024-03-24T11:47:58.088 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:44:32.375573439 +0000 2024-03-24T11:47:58.088 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:44:32.375573439 +0000 2024-03-24T11:47:58.088 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:44:32.367573594 +0000 2024-03-24T11:47:58.088 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-24T11:47:58.154 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:58 smithi189 systemd[1]: Starting Ceph mgr.z for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:47:58.157 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:47:58.157 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:47:58.157 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000239984 s, 2.1 MB/s 2024-03-24T11:47:58.159 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-24T11:47:58.214 INFO:tasks.cephadm:Deploying osd.0 on smithi130 with /dev/nvme4n1... 2024-03-24T11:47:58.214 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-24T11:47:58.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:58 smithi189 ceph-mon[30629]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:58.413 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:58 smithi189 podman[32697]: 2024-03-24 11:47:58.175050524 +0000 UTC m=+0.020149002 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:47:58.413 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:58 smithi189 podman[32697]: 2024-03-24 11:47:58.290055702 +0000 UTC m=+0.135154168 container create 78ee45e0c014da72a244eb0dfb735f24f197d0ff3e90cc4bea2c3c0f2137e5e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T11:47:58.457 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:47:58.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:58 smithi130 ceph-mon[25795]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:58.739 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:58 smithi174 ceph-mon[30177]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:47:58.946 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:58 smithi189 podman[32697]: 2024-03-24 11:47:58.509019802 +0000 UTC m=+0.354118268 container init 78ee45e0c014da72a244eb0dfb735f24f197d0ff3e90cc4bea2c3c0f2137e5e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T11:47:58.946 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:58 smithi189 podman[32697]: 2024-03-24 11:47:58.514504286 +0000 UTC m=+0.359602752 container start 78ee45e0c014da72a244eb0dfb735f24f197d0ff3e90cc4bea2c3c0f2137e5e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD) 2024-03-24T11:47:58.946 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:58 smithi189 ceph-mgr[32763]: -- 172.21.15.189:0/1451950527 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555fe5cc21a0 con 0x555fe5c98800 2024-03-24T11:47:58.946 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:58 smithi189 bash[32697]: 78ee45e0c014da72a244eb0dfb735f24f197d0ff3e90cc4bea2c3c0f2137e5e3 2024-03-24T11:47:58.946 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:58 smithi189 systemd[1]: Started Ceph mgr.z for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:47:59.446 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:47:59.075+0000 7f18ebad6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:47:59.446 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:47:59.170+0000 7f18ebad6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:47:59.842 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:59 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:59.842 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:59 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:59.842 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:59 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:47:59.842 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:59 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:47:59.843 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:59 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:47:59.843 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:59 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:47:59.843 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:59 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:47:59.843 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:59 smithi130 ceph-mon[25795]: Deploying daemon mgr.x on smithi174 2024-03-24T11:47:59.843 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:47:59 smithi130 ceph-mon[25795]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:00.048 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:00.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:00.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:00.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:00.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:48:00.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:00.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:00.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-mon[30629]: Deploying daemon mgr.x on smithi174 2024-03-24T11:48:00.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-mon[30629]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:00.049 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:47:59.815+0000 7f18ebad6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:48:00.049 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:47:59 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:47:59.907+0000 7f18ebad6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:48:00.050 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:00 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:00.047+0000 7f18ebad6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:48:00.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:59 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:00.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:59 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:00.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:59 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:00.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:59 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:00.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:59 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:48:00.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:59 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:00.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:59 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:00.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:59 smithi174 ceph-mon[30177]: Deploying daemon mgr.x on smithi174 2024-03-24T11:48:00.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:47:59 smithi174 ceph-mon[30177]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:00.408 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/nvme4n1 2024-03-24T11:48:00.408 INFO:teuthology.orchestra.run.smithi130.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-24T11:48:00.408 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-24T11:48:00.408 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-03-24T11:48:00.408 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-03-24T11:48:00.408 INFO:teuthology.orchestra.run.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122613 s, 855 MB/s 2024-03-24T11:48:00.408 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-03-24T11:48:00.445 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:00 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:00.143+0000 7f18ebad6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:48:00.446 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:00 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:00.225+0000 7f18ebad6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:48:00.775 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:00 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:00.488+0000 7f18ebad6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:48:00.775 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:00 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:00.596+0000 7f18ebad6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:48:00.775 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:00 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:00.681+0000 7f18ebad6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:48:00.776 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:00 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:00.774+0000 7f18ebad6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:48:00.784 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:00 smithi174 systemd[1]: Starting Ceph mgr.x for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:48:00.994 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi130:/dev/nvme4n1 2024-03-24T11:48:01.094 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:00 smithi174 podman[31630]: 2024-03-24 11:48:00.999073537 +0000 UTC m=+0.115070134 container create 3909890b4ca3e23dba86ba0cc4e98a579ffd1735add610c8b1d33cf58caa9fa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T11:48:01.094 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:00 smithi174 podman[31630]: 2024-03-24 11:48:00.902784975 +0000 UTC m=+0.018781585 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:48:01.195 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:00 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:00.950+0000 7f18ebad6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:48:01.228 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:48:01.594 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:01 smithi174 podman[31630]: 2024-03-24 11:48:01.209489047 +0000 UTC m=+0.325485658 container init 3909890b4ca3e23dba86ba0cc4e98a579ffd1735add610c8b1d33cf58caa9fa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-24T11:48:01.594 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:01 smithi174 podman[31630]: 2024-03-24 11:48:01.21455722 +0000 UTC m=+0.330553819 container start 3909890b4ca3e23dba86ba0cc4e98a579ffd1735add610c8b1d33cf58caa9fa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T11:48:01.594 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:01 smithi174 ceph-mgr[31668]: -- 172.21.15.174:0/1587560885 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c2f51e21a0 con 0x55c2f51b9000 2024-03-24T11:48:01.594 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:01 smithi174 bash[31630]: 3909890b4ca3e23dba86ba0cc4e98a579ffd1735add610c8b1d33cf58caa9fa1 2024-03-24T11:48:01.594 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:01 smithi174 systemd[1]: Started Ceph mgr.x for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:48:02.094 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:01 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:01.779+0000 7f2b71887200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:48:02.094 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:01 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:01.875+0000 7f2b71887200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:48:02.245 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:01 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:01.951+0000 7f18ebad6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:48:02.245 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:02 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:02.096+0000 7f18ebad6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:48:02.470 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:02 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.470 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:02 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.470 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:02 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.470 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:02 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.470 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:02 smithi130 ceph-mon[25795]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:02.470 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:02 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:02.533 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:02 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.533 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:02 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.533 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:02 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.533 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:02 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.533 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:02 smithi189 ceph-mon[30629]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:02.533 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:02 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:02.534 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:02 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:02.441+0000 7f18ebad6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:48:02.534 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:02 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:02.532+0000 7f18ebad6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:48:02.564 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:02 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:02.563+0000 7f2b71887200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:48:02.564 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:02 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.564 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:02 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.564 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:02 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.564 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:02 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:02.564 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:02 smithi174 ceph-mon[30177]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:02.564 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:02 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:02.844 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:02 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:02.656+0000 7f2b71887200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:48:02.844 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:02 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:02.806+0000 7f2b71887200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:48:02.923 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:02 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:02.737+0000 7f18ebad6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:48:02.923 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:02 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:02.922+0000 7f18ebad6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:48:03.249 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:02 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:02.904+0000 7f2b71887200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:48:03.249 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:02 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:02.986+0000 7f2b71887200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:48:03.249 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:03.248+0000 7f2b71887200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:48:03.463 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:03.205+0000 7f18ebad6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:48:03.540 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:03.357+0000 7f2b71887200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:48:03.540 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:03.443+0000 7f2b71887200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:48:03.540 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:03.538+0000 7f2b71887200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:48:03.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:03.540 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:03.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:03.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:03.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:03.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:03.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:03.541 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:03.745 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.745 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.745 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.745 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.746 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.746 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:03.746 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:03.746 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.746 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:03.746 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:03.746 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:03.746 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:03.746 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:03.746 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:03 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:03.844 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:03 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:03.715+0000 7f2b71887200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:48:03.869 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:03.466+0000 7f18ebad6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:48:03.869 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:03.699+0000 7f18ebad6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:48:03.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:03.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:03.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:03.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:48:03.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:48:03.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:03.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:03.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:03.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:03 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:04.692 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:04 smithi130 ceph-mon[25795]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:48:04.692 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:04 smithi130 ceph-mon[25795]: Reconfiguring daemon mgr.y on smithi130 2024-03-24T11:48:04.692 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:04 smithi130 ceph-mon[25795]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.692 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:04 smithi130 ceph-mon[25795]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:04.817 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:04 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:04.684+0000 7f2b71887200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:48:04.817 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:04 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:04.816+0000 7f2b71887200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:48:04.817 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:04 smithi174 ceph-mon[30177]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:48:04.818 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:04 smithi174 ceph-mon[30177]: Reconfiguring daemon mgr.y on smithi130 2024-03-24T11:48:04.818 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:04 smithi174 ceph-mon[30177]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.818 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:04 smithi174 ceph-mon[30177]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:04.945 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:04 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:04.499+0000 7f18ebad6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:48:04.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:04 smithi189 ceph-mon[30629]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:48:04.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:04 smithi189 ceph-mon[30629]: Reconfiguring daemon mgr.y on smithi130 2024-03-24T11:48:04.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:04 smithi189 ceph-mon[30629]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:04.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:04 smithi189 ceph-mon[30629]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:05.417 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:05.105+0000 7f18ebad6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:48:05.418 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:05.203+0000 7f18ebad6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:48:05.418 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:48:05.416+0000 7f18ebad6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:48:05.432 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:05.151+0000 7f2b71887200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:48:05.432 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:05.239+0000 7f2b71887200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:48:05.432 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:05.431+0000 7f2b71887200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:48:05.752 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:05.614+0000 7f2b71887200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:48:06.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:06.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:06.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:06.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:06.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:06.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:06.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:06.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: Standby manager daemon z started 2024-03-24T11:48:06.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:48:06.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:06.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:48:06.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:05 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:06.094 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:05.897+0000 7f2b71887200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:48:06.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:06.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:06.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:06.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:06.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:06.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:06.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:06.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: Standby manager daemon z started 2024-03-24T11:48:06.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:48:06.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:06.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:48:06.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:05 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:06.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:06.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:06.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:06.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:06.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:06.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:06.197 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:06.197 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: Standby manager daemon z started 2024-03-24T11:48:06.197 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:48:06.197 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:06.197 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:48:06.197 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:05 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.189:0/3005537701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:06.594 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:06 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:06.159+0000 7f2b71887200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:48:06.594 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:06 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:06.392+0000 7f2b71887200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:48:07.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:06 smithi174 ceph-mon[30177]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-24T11:48:07.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:06 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:48:07.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:06 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/353819112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d673100d-5f3a-458b-8800-a45d87c7b0c2"}]: dispatch 2024-03-24T11:48:07.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:06 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/353819112' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d673100d-5f3a-458b-8800-a45d87c7b0c2"}]': finished 2024-03-24T11:48:07.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:06 smithi174 ceph-mon[30177]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:48:07.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:06 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:07.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:06 smithi189 ceph-mon[30629]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-24T11:48:07.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:06 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:48:07.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:06 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/353819112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d673100d-5f3a-458b-8800-a45d87c7b0c2"}]: dispatch 2024-03-24T11:48:07.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:06 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/353819112' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d673100d-5f3a-458b-8800-a45d87c7b0c2"}]': finished 2024-03-24T11:48:07.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:06 smithi189 ceph-mon[30629]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:48:07.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:06 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:07.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:06 smithi130 ceph-mon[25795]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-24T11:48:07.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:06 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:48:07.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:06 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/353819112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d673100d-5f3a-458b-8800-a45d87c7b0c2"}]: dispatch 2024-03-24T11:48:07.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:06 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/353819112' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d673100d-5f3a-458b-8800-a45d87c7b0c2"}]': finished 2024-03-24T11:48:07.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:06 smithi130 ceph-mon[25795]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:48:07.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:06 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:07.594 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:07 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:07.183+0000 7f2b71887200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:48:08.063 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:07 smithi174 ceph-mon[30177]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:08.063 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:07 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:07.752+0000 7f2b71887200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:48:08.063 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:07 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:07.850+0000 7f2b71887200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:48:08.063 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:48:08 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:48:08.061+0000 7f2b71887200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:48:08.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:07 smithi189 ceph-mon[30629]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:07 smithi130 ceph-mon[25795]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:09.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:08 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/1947872184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:09.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:08 smithi174 ceph-mon[30177]: Standby manager daemon x started 2024-03-24T11:48:09.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:08 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:48:09.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:08 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:09.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:08 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:48:09.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:08 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:09.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:08 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/1947872184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:09.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:08 smithi189 ceph-mon[30629]: Standby manager daemon x started 2024-03-24T11:48:09.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:08 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:48:09.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:08 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:09.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:08 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:48:09.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:08 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:09.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:08 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/1947872184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:09.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:08 smithi130 ceph-mon[25795]: Standby manager daemon x started 2024-03-24T11:48:09.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:08 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:48:09.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:08 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:48:09.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:08 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:48:09.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:08 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.174:0/1461528543' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:48:10.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:09 smithi174 ceph-mon[30177]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-24T11:48:10.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:09 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:48:10.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:09 smithi174 ceph-mon[30177]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:10.101 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:09 smithi130 ceph-mon[25795]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-24T11:48:10.101 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:09 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:48:10.101 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:09 smithi130 ceph-mon[25795]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:10.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:09 smithi189 ceph-mon[30629]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-24T11:48:10.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:09 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:48:10.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:09 smithi189 ceph-mon[30629]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:12.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:12 smithi130 ceph-mon[25795]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:12.845 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:12 smithi174 ceph-mon[30177]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:12.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:12 smithi189 ceph-mon[30629]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:14.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:14 smithi130 ceph-mon[25795]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:14.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:14 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:48:14.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:14 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:14.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:14 smithi174 ceph-mon[30177]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:14.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:14 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:48:14.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:14 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:14.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:14 smithi189 ceph-mon[30629]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:14.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:14 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:48:14.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:14 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:15.712 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:15 smithi130 ceph-mon[25795]: Deploying daemon osd.0 on smithi130 2024-03-24T11:48:15.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:15 smithi174 ceph-mon[30177]: Deploying daemon osd.0 on smithi130 2024-03-24T11:48:15.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:15 smithi189 ceph-mon[30629]: Deploying daemon osd.0 on smithi130 2024-03-24T11:48:16.765 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:16 smithi130 ceph-mon[25795]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:16.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:16 smithi174 ceph-mon[30177]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:16.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:16 smithi189 ceph-mon[30629]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:18.498 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:18 smithi130 ceph-mon[25795]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:18.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:18 smithi174 ceph-mon[30177]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:18.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:18 smithi189 ceph-mon[30629]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.063 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:20.063 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:20.063 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:20.063 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:20 smithi130 ceph-mon[25795]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:20.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:20.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:20.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:20 smithi174 ceph-mon[30177]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:20.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:20.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:20.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:20.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:20 smithi189 ceph-mon[30629]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:22.465 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 0 on host 'smithi130' 2024-03-24T11:48:22.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:22 smithi130 ceph-mon[25795]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:22.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:22 smithi130 ceph-mon[25795]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:48:22.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:22 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:22.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:22 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:22.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:22 smithi174 ceph-mon[30177]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:22.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:22 smithi174 ceph-mon[30177]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:48:22.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:22 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:22.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:22 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:22.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:22 smithi189 ceph-mon[30629]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:22.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:22 smithi189 ceph-mon[30629]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:48:22.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:22 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:22.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:22 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:23.113 DEBUG:teuthology.orchestra.run.smithi130:osd.0> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T11:48:23.115 INFO:tasks.cephadm:Deploying osd.1 on smithi174 with /dev/nvme4n1... 2024-03-24T11:48:23.115 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-24T11:48:23.355 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:48:23.650 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:23 smithi130 ceph-mon[25795]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:48:23.650 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:23 smithi130 ceph-mon[25795]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:48:23.650 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:23 smithi130 ceph-mon[25795]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-24T11:48:23.650 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:23 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:23.650 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:23 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:23.650 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:23 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:23.650 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:23 smithi130 ceph-mon[25795]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-24T11:48:23.650 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:23 smithi130 ceph-mon[25795]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:48:23.650 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:23 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:23.651 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:48:23 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0[40578]: 2024-03-24T11:48:23.469+0000 7fb32a258700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T11:48:23.651 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:48:23 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0[40578]: 2024-03-24T11:48:23.474+0000 7fb32604b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:48:23.764 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:23 smithi174 ceph-mon[30177]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:48:23.764 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:23 smithi174 ceph-mon[30177]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:48:23.764 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:23 smithi174 ceph-mon[30177]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-24T11:48:23.764 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:23 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:23.764 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:23 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:23.764 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:23 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:23.764 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:23 smithi174 ceph-mon[30177]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-24T11:48:23.764 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:23 smithi174 ceph-mon[30177]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:48:23.764 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:23 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:23.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:23 smithi189 ceph-mon[30629]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:48:23.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:23 smithi189 ceph-mon[30629]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:48:23.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:23 smithi189 ceph-mon[30629]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-24T11:48:23.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:23 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:23.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:23 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:23.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:23 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:23.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:23 smithi189 ceph-mon[30629]: from='osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-24T11:48:23.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:23 smithi189 ceph-mon[30629]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:48:23.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:23 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:24.531 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:24 smithi130 ceph-mon[25795]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:24.531 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:24 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:24.531 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:24 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:24.780 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:24 smithi174 ceph-mon[30177]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:24.780 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:24 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:24.780 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:24 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:24.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:24 smithi189 ceph-mon[30629]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:24.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:24 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:24.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:24 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:25.194 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/nvme4n1 2024-03-24T11:48:25.194 INFO:teuthology.orchestra.run.smithi174.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-24T11:48:25.194 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-24T11:48:25.194 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-03-24T11:48:25.194 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-03-24T11:48:25.194 INFO:teuthology.orchestra.run.smithi174.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124608 s, 841 MB/s 2024-03-24T11:48:25.194 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-03-24T11:48:25.741 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:25 smithi174 ceph-mon[30177]: purged_snaps scrub starts 2024-03-24T11:48:25.741 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:25 smithi174 ceph-mon[30177]: purged_snaps scrub ok 2024-03-24T11:48:25.741 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:25 smithi174 ceph-mon[30177]: osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797] boot 2024-03-24T11:48:25.741 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:25 smithi174 ceph-mon[30177]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:48:25.741 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:25 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:25.754 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:25 smithi130 ceph-mon[25795]: purged_snaps scrub starts 2024-03-24T11:48:25.754 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:25 smithi130 ceph-mon[25795]: purged_snaps scrub ok 2024-03-24T11:48:25.754 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:25 smithi130 ceph-mon[25795]: osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797] boot 2024-03-24T11:48:25.754 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:25 smithi130 ceph-mon[25795]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:48:25.754 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:25.769 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi174:/dev/nvme4n1 2024-03-24T11:48:25.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:25 smithi189 ceph-mon[30629]: purged_snaps scrub starts 2024-03-24T11:48:25.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:25 smithi189 ceph-mon[30629]: purged_snaps scrub ok 2024-03-24T11:48:25.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:25 smithi189 ceph-mon[30629]: osd.0 [v2:172.21.15.130:6802/2720668797,v1:172.21.15.130:6803/2720668797] boot 2024-03-24T11:48:25.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:25 smithi189 ceph-mon[30629]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:48:25.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:25 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:48:26.002 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:48:26.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:26 smithi130 ceph-mon[25795]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:26.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:26 smithi130 ceph-mon[25795]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:48:26.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:26 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:26.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:26 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:26.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:26 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:48:26.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:26 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:26.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:26 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:26.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:26 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:26.539 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:26 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:26.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:26 smithi174 ceph-mon[30177]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:26.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:26 smithi174 ceph-mon[30177]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:48:26.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:26 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:26.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:26 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:26.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:26 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:48:26.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:26 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:26.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:26 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:26.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:26 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:26.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:26 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:26.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:26 smithi189 ceph-mon[30629]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:48:26.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:26 smithi189 ceph-mon[30629]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:48:26.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:26 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:26.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:26 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:26.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:26 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:48:26.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:26 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:26.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:26 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:26.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:26 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:26.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:26 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:27.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:27 smithi130 ceph-mon[25795]: Detected new or changed devices on smithi130 2024-03-24T11:48:27.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:27 smithi130 ceph-mon[25795]: Adjusting osd_memory_target on smithi130 to 17088M 2024-03-24T11:48:27.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:27 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:27.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:27 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:27.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:27 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:27.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:27 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:27.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:27 smithi174 ceph-mon[30177]: Detected new or changed devices on smithi130 2024-03-24T11:48:27.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:27 smithi174 ceph-mon[30177]: Adjusting osd_memory_target on smithi130 to 17088M 2024-03-24T11:48:27.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:27 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:27.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:27 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:27.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:27 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:27.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:27 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:27.847 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:27 smithi189 ceph-mon[30629]: Detected new or changed devices on smithi130 2024-03-24T11:48:27.847 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:27 smithi189 ceph-mon[30629]: Adjusting osd_memory_target on smithi130 to 17088M 2024-03-24T11:48:27.847 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:27 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:27.847 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:27 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:27.847 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:27 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:27.847 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:27 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:28.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:28 smithi189 ceph-mon[30629]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:28.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:28 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:28.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:28 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:28.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:28 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:28.745 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:28 smithi174 ceph-mon[30177]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:28.745 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:28 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:28.745 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:28 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:28.745 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:28 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:28 smithi130 ceph-mon[25795]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:28 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:28 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:28.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:28 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:29.753 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:29 smithi174 ceph-mon[30177]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:29.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:29 smithi130 ceph-mon[25795]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:29.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:29 smithi189 ceph-mon[30629]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:30.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:30 smithi130 ceph-mon[25795]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:30.802 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:30 smithi174 ceph-mon[30177]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:30.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:30 smithi189 ceph-mon[30629]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:31.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:31 smithi130 ceph-mon[25795]: from='client.? 172.21.15.174:0/4197176793' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d623983-bbdf-4791-95e6-f1465b99480c"}]: dispatch 2024-03-24T11:48:31.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:31 smithi130 ceph-mon[25795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d623983-bbdf-4791-95e6-f1465b99480c"}]: dispatch 2024-03-24T11:48:31.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:31 smithi130 ceph-mon[25795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d623983-bbdf-4791-95e6-f1465b99480c"}]': finished 2024-03-24T11:48:31.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:31 smithi130 ceph-mon[25795]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:48:31.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:31 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:31.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:31 smithi174 ceph-mon[30177]: from='client.? 172.21.15.174:0/4197176793' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d623983-bbdf-4791-95e6-f1465b99480c"}]: dispatch 2024-03-24T11:48:31.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:31 smithi174 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d623983-bbdf-4791-95e6-f1465b99480c"}]: dispatch 2024-03-24T11:48:31.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:31 smithi174 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d623983-bbdf-4791-95e6-f1465b99480c"}]': finished 2024-03-24T11:48:31.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:31 smithi174 ceph-mon[30177]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:48:31.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:31 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:31.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:31 smithi189 ceph-mon[30629]: from='client.? 172.21.15.174:0/4197176793' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d623983-bbdf-4791-95e6-f1465b99480c"}]: dispatch 2024-03-24T11:48:31.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:31 smithi189 ceph-mon[30629]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d623983-bbdf-4791-95e6-f1465b99480c"}]: dispatch 2024-03-24T11:48:31.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:31 smithi189 ceph-mon[30629]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d623983-bbdf-4791-95e6-f1465b99480c"}]': finished 2024-03-24T11:48:31.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:31 smithi189 ceph-mon[30629]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:48:31.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:31 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:32.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:32 smithi130 ceph-mon[25795]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:32.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:32 smithi130 ceph-mon[25795]: from='client.? 172.21.15.174:0/197922955' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:32.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:32 smithi174 ceph-mon[30177]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:32.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:32 smithi174 ceph-mon[30177]: from='client.? 172.21.15.174:0/197922955' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:32.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:32 smithi189 ceph-mon[30629]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:32.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:32 smithi189 ceph-mon[30629]: from='client.? 172.21.15.174:0/197922955' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:34.699 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:34 smithi174 ceph-mon[30177]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:34.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:34 smithi130 ceph-mon[25795]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:34.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:34 smithi189 ceph-mon[30629]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:36.543 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:36 smithi174 ceph-mon[30177]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:36.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:36 smithi130 ceph-mon[25795]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:36.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:36 smithi189 ceph-mon[30629]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:38.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:38 smithi130 ceph-mon[25795]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:38.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:38 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:48:38.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:38 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:38.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:38 smithi130 ceph-mon[25795]: Deploying daemon osd.1 on smithi174 2024-03-24T11:48:38.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:38 smithi174 ceph-mon[30177]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:38.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:38 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:48:38.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:38 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:38.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:38 smithi174 ceph-mon[30177]: Deploying daemon osd.1 on smithi174 2024-03-24T11:48:38.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:38 smithi189 ceph-mon[30629]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:38.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:38 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:48:38.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:38 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:38.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:38 smithi189 ceph-mon[30629]: Deploying daemon osd.1 on smithi174 2024-03-24T11:48:40.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:40 smithi174 ceph-mon[30177]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:40.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:40 smithi130 ceph-mon[25795]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:40.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:40 smithi189 ceph-mon[30629]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:42.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:42 smithi174 ceph-mon[30177]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:42.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:42 smithi130 ceph-mon[25795]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:42.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:42 smithi189 ceph-mon[30629]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:44.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:44 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:44.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:44 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:44.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:44 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:44.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:44 smithi189 ceph-mon[30629]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:44.536 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:44 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:44.536 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:44 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:44.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:44 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:44.537 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:44 smithi174 ceph-mon[30177]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:44.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:44 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:44.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:44 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:44.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:44 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:44.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:44 smithi130 ceph-mon[25795]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:46.546 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:46 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:46.546 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:46 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:46.546 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:46 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:46.546 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:46 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:46.546 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:46 smithi174 ceph-mon[30177]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:46.546 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:46 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:46.546 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:46 smithi174 ceph-mon[30177]: from='osd.1 [v2:172.21.15.174:6800/3224679846,v1:172.21.15.174:6801/3224679846]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:48:46.546 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:46 smithi174 ceph-mon[30177]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:48:46.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:46.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:46.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:46.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:46.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:46 smithi189 ceph-mon[30629]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:46.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:46 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:46.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:46 smithi189 ceph-mon[30629]: from='osd.1 [v2:172.21.15.174:6800/3224679846,v1:172.21.15.174:6801/3224679846]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:48:46.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:46 smithi189 ceph-mon[30629]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:48:46.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:46.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:46.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:46.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:46.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:46 smithi130 ceph-mon[25795]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:46.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:46 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:46.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:46 smithi130 ceph-mon[25795]: from='osd.1 [v2:172.21.15.174:6800/3224679846,v1:172.21.15.174:6801/3224679846]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:48:46.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:46 smithi130 ceph-mon[25795]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:48:46.916 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 1 on host 'smithi174' 2024-03-24T11:48:47.465 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:47 smithi174 ceph-mon[30177]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:48:47.465 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:47 smithi174 ceph-mon[30177]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:48:47.466 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:47 smithi174 ceph-mon[30177]: from='osd.1 [v2:172.21.15.174:6800/3224679846,v1:172.21.15.174:6801/3224679846]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-24T11:48:47.466 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:47 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:47.466 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:47 smithi174 ceph-mon[30177]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-24T11:48:47.466 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:47 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:47.466 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:47 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:47.466 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:47 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:47.491 DEBUG:teuthology.orchestra.run.smithi174:osd.1> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T11:48:47.492 INFO:tasks.cephadm:Deploying osd.2 on smithi189 with /dev/nvme4n1... 2024-03-24T11:48:47.493 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-24T11:48:47.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:47 smithi189 ceph-mon[30629]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:48:47.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:47 smithi189 ceph-mon[30629]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:48:47.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:47 smithi189 ceph-mon[30629]: from='osd.1 [v2:172.21.15.174:6800/3224679846,v1:172.21.15.174:6801/3224679846]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-24T11:48:47.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:47 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:47.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:47 smithi189 ceph-mon[30629]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-24T11:48:47.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:47 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:47.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:47 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:47.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:47 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:47.739 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:48:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:47 smithi130 ceph-mon[25795]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:48:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:47 smithi130 ceph-mon[25795]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:48:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:47 smithi130 ceph-mon[25795]: from='osd.1 [v2:172.21.15.174:6800/3224679846,v1:172.21.15.174:6801/3224679846]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-24T11:48:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:47 smithi130 ceph-mon[25795]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-03-24T11:48:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:48:47.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:47 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:48.654 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:48 smithi174 ceph-mon[30177]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:48.655 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:48 smithi174 ceph-mon[30177]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-24T11:48:48.655 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:48 smithi174 ceph-mon[30177]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:48:48.655 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:48.655 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:48 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:48.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:48 smithi189 ceph-mon[30629]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:48.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:48 smithi189 ceph-mon[30629]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-24T11:48:48.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:48 smithi189 ceph-mon[30629]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:48:48.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:48 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:48.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:48 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:48.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:48 smithi130 ceph-mon[25795]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:48.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:48 smithi130 ceph-mon[25795]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-03-24T11:48:48.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:48 smithi130 ceph-mon[25795]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:48:48.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:48 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:48.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:48 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:49.650 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/nvme4n1 2024-03-24T11:48:49.651 INFO:teuthology.orchestra.run.smithi189.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-24T11:48:49.651 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-24T11:48:49.651 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-03-24T11:48:49.651 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-03-24T11:48:49.651 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123932 s, 846 MB/s 2024-03-24T11:48:49.651 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-03-24T11:48:49.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:49 smithi189 ceph-mon[30629]: purged_snaps scrub starts 2024-03-24T11:48:49.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:49 smithi189 ceph-mon[30629]: purged_snaps scrub ok 2024-03-24T11:48:49.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:49 smithi189 ceph-mon[30629]: osd.1 [v2:172.21.15.174:6800/3224679846,v1:172.21.15.174:6801/3224679846] boot 2024-03-24T11:48:49.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:49 smithi189 ceph-mon[30629]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:48:49.696 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:49.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:49 smithi130 ceph-mon[25795]: purged_snaps scrub starts 2024-03-24T11:48:49.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:49 smithi130 ceph-mon[25795]: purged_snaps scrub ok 2024-03-24T11:48:49.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:49 smithi130 ceph-mon[25795]: osd.1 [v2:172.21.15.174:6800/3224679846,v1:172.21.15.174:6801/3224679846] boot 2024-03-24T11:48:49.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:49 smithi130 ceph-mon[25795]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:48:49.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:49.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:49 smithi174 ceph-mon[30177]: purged_snaps scrub starts 2024-03-24T11:48:49.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:49 smithi174 ceph-mon[30177]: purged_snaps scrub ok 2024-03-24T11:48:49.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:49 smithi174 ceph-mon[30177]: osd.1 [v2:172.21.15.174:6800/3224679846,v1:172.21.15.174:6801/3224679846] boot 2024-03-24T11:48:49.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:49 smithi174 ceph-mon[30177]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:48:49.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:48:50.247 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi189:/dev/nvme4n1 2024-03-24T11:48:50.480 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:48:50.603 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:50 smithi189 ceph-mon[30629]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:50.603 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:50 smithi189 ceph-mon[30629]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:48:50.727 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:50 smithi174 ceph-mon[30177]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:50.727 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:50 smithi174 ceph-mon[30177]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:48:50.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:50 smithi130 ceph-mon[25795]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:48:50.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:50 smithi130 ceph-mon[25795]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:48:52.061 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:51 smithi189 ceph-mon[30629]: Detected new or changed devices on smithi174 2024-03-24T11:48:52.061 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:51 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.061 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:51 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.061 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:51 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:48:52.061 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:51 smithi189 ceph-mon[30629]: Adjusting osd_memory_target on smithi174 to 17088M 2024-03-24T11:48:52.061 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:51 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.062 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:51 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:52.062 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:51 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:52.062 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:51 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.062 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:51 smithi189 ceph-mon[30629]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:48:52.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:51 smithi174 ceph-mon[30177]: Detected new or changed devices on smithi174 2024-03-24T11:48:52.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:51 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:51 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:51 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:48:52.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:51 smithi174 ceph-mon[30177]: Adjusting osd_memory_target on smithi174 to 17088M 2024-03-24T11:48:52.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:51 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:51 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:52.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:51 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:52.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:51 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:51 smithi174 ceph-mon[30177]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:48:52.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:51 smithi130 ceph-mon[25795]: Detected new or changed devices on smithi174 2024-03-24T11:48:52.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:51 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:51 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:51 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:48:52.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:51 smithi130 ceph-mon[25795]: Adjusting osd_memory_target on smithi174 to 17088M 2024-03-24T11:48:52.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:51 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.289 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:51 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:52.289 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:51 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:48:52.289 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:51 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:48:52.289 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:51 smithi130 ceph-mon[25795]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:48:53.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:52 smithi174 ceph-mon[30177]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:53.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:52 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:53.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:52 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:53.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:52 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:53.096 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:52 smithi189 ceph-mon[30629]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:53.096 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:52 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:53.096 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:52 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:53.096 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:52 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:53.190 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:52 smithi130 ceph-mon[25795]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:48:53.190 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:52 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:48:53.190 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:52 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:48:53.190 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:52 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:48:53.941 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:53 smithi189 ceph-mon[30629]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:48:54.037 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:53 smithi130 ceph-mon[25795]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:48:54.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:53 smithi174 ceph-mon[30177]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:48:55.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:55 smithi189 ceph-mon[30629]: from='client.? 172.21.15.189:0/1092129508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25926a0b-b35b-4d4d-9c4f-ecaf374ee267"}]: dispatch 2024-03-24T11:48:55.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:55 smithi189 ceph-mon[30629]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25926a0b-b35b-4d4d-9c4f-ecaf374ee267"}]: dispatch 2024-03-24T11:48:55.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:55 smithi189 ceph-mon[30629]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25926a0b-b35b-4d4d-9c4f-ecaf374ee267"}]': finished 2024-03-24T11:48:55.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:55 smithi189 ceph-mon[30629]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:48:55.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:55 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:48:55.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:55 smithi130 ceph-mon[25795]: from='client.? 172.21.15.189:0/1092129508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25926a0b-b35b-4d4d-9c4f-ecaf374ee267"}]: dispatch 2024-03-24T11:48:55.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:55 smithi130 ceph-mon[25795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25926a0b-b35b-4d4d-9c4f-ecaf374ee267"}]: dispatch 2024-03-24T11:48:55.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:55 smithi130 ceph-mon[25795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25926a0b-b35b-4d4d-9c4f-ecaf374ee267"}]': finished 2024-03-24T11:48:55.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:55 smithi130 ceph-mon[25795]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:48:55.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:55 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:48:55.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:55 smithi174 ceph-mon[30177]: from='client.? 172.21.15.189:0/1092129508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25926a0b-b35b-4d4d-9c4f-ecaf374ee267"}]: dispatch 2024-03-24T11:48:55.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:55 smithi174 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25926a0b-b35b-4d4d-9c4f-ecaf374ee267"}]: dispatch 2024-03-24T11:48:55.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:55 smithi174 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25926a0b-b35b-4d4d-9c4f-ecaf374ee267"}]': finished 2024-03-24T11:48:55.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:55 smithi174 ceph-mon[30177]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:48:55.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:55 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:48:56.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:56 smithi189 ceph-mon[30629]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:48:56.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:56 smithi130 ceph-mon[25795]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:48:56.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:56 smithi174 ceph-mon[30177]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:48:57.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:57 smithi189 ceph-mon[30629]: from='client.? 172.21.15.189:0/3629027275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:57.537 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:57 smithi130 ceph-mon[25795]: from='client.? 172.21.15.189:0/3629027275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:57.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:57 smithi174 ceph-mon[30177]: from='client.? 172.21.15.189:0/3629027275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:48:58.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:48:58 smithi189 ceph-mon[30629]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:48:58.495 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:48:58 smithi174 ceph-mon[30177]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:48:58.537 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:48:58 smithi130 ceph-mon[25795]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:00.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:00 smithi130 ceph-mon[25795]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:00.832 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:00 smithi189 ceph-mon[30629]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:00.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:00 smithi174 ceph-mon[30177]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:02.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:02 smithi130 ceph-mon[25795]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:02.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:02 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:49:02.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:02 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:02.827 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:02 smithi189 ceph-mon[30629]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:02.827 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:02 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:49:02.827 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:02 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:02.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:02 smithi174 ceph-mon[30177]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:02.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:02 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:49:02.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:02 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:03.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:03 smithi130 ceph-mon[25795]: Deploying daemon osd.2 on smithi189 2024-03-24T11:49:03.826 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:03 smithi189 ceph-mon[30629]: Deploying daemon osd.2 on smithi189 2024-03-24T11:49:03.843 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:03 smithi174 ceph-mon[30177]: Deploying daemon osd.2 on smithi189 2024-03-24T11:49:04.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:04 smithi130 ceph-mon[25795]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:04.833 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:04 smithi189 ceph-mon[30629]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:04.843 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:04 smithi174 ceph-mon[30177]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:06.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:06 smithi130 ceph-mon[25795]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:06.843 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:06 smithi174 ceph-mon[30177]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:06.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:06 smithi189 ceph-mon[30629]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:08.754 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:08 smithi189 ceph-mon[30629]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:08.754 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:08 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:08.754 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:08 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:08.754 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:08 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:08.787 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:08 smithi130 ceph-mon[25795]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:08.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:08 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:08.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:08 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:08.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:08 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:08.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:08 smithi174 ceph-mon[30177]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:08.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:08 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:08.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:08 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:08.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:08 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:10.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:10 smithi189 ceph-mon[30629]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:10.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:10 smithi130 ceph-mon[25795]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:10.843 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:10 smithi174 ceph-mon[30177]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:11.188 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 2 on host 'smithi189' 2024-03-24T11:49:11.811 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:11 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:11.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:11 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:11.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:11 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:11.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:11 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:11.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:11 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:11.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:11 smithi189 ceph-mon[30629]: from='osd.2 [v2:172.21.15.189:6800/627265002,v1:172.21.15.189:6801/627265002]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:11.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:11 smithi189 ceph-mon[30629]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:11.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:11 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:11.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:11 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:11.813 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:11 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:11.813 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:11 smithi189 ceph-mon[30629]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:11.842 DEBUG:teuthology.orchestra.run.smithi189:osd.2> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T11:49:11.844 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-24T11:49:11.845 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T11:49:11.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:11 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:11.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:11 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:11.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:11 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:11.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:11 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:11.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:11 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:11.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:11 smithi130 ceph-mon[25795]: from='osd.2 [v2:172.21.15.189:6800/627265002,v1:172.21.15.189:6801/627265002]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:11.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:11 smithi130 ceph-mon[25795]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:11.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:11 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:11.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:11 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:11.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:11 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:11.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:11 smithi130 ceph-mon[25795]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:12.091 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:12.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:11 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:12.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:11 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:12.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:11 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:12.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:11 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:12.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:11 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:12.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:11 smithi174 ceph-mon[30177]: from='osd.2 [v2:172.21.15.189:6800/627265002,v1:172.21.15.189:6801/627265002]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:12.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:11 smithi174 ceph-mon[30177]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:49:12.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:11 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:12.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:11 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:12.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:11 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:12.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:11 smithi174 ceph-mon[30177]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:12.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:12 smithi130 ceph-mon[25795]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:49:12.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:12 smithi130 ceph-mon[25795]: from='osd.2 [v2:172.21.15.189:6800/627265002,v1:172.21.15.189:6801/627265002]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:49:12.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:12 smithi130 ceph-mon[25795]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:49:12.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:12 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:12.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:12 smithi130 ceph-mon[25795]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:49:12.817 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:12 smithi189 ceph-mon[30629]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:49:12.817 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:12 smithi189 ceph-mon[30629]: from='osd.2 [v2:172.21.15.189:6800/627265002,v1:172.21.15.189:6801/627265002]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:49:12.817 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:12 smithi189 ceph-mon[30629]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:49:12.817 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:12 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:12.817 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:12 smithi189 ceph-mon[30629]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:49:12.818 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:49:12 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2[35929]: 2024-03-24T11:49:12.640+0000 7f5defadf700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T11:49:12.818 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:49:12 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2[35929]: 2024-03-24T11:49:12.643+0000 7f5deb8d2700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:49:13.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:12 smithi174 ceph-mon[30177]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:49:13.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:12 smithi174 ceph-mon[30177]: from='osd.2 [v2:172.21.15.189:6800/627265002,v1:172.21.15.189:6801/627265002]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:49:13.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:12 smithi174 ceph-mon[30177]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:49:13.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:12 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:13.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:12 smithi174 ceph-mon[30177]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:49:13.839 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:49:13.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:13 smithi189 ceph-mon[30629]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T11:49:13.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:13 smithi189 ceph-mon[30629]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:49:13.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:13 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:13.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:13 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:13.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:13 smithi189 ceph-mon[30629]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:14.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:13 smithi130 ceph-mon[25795]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T11:49:14.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:13 smithi130 ceph-mon[25795]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:49:14.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:13 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:14.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:13 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:14.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:13 smithi130 ceph-mon[25795]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:14.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:13 smithi174 ceph-mon[30177]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T11:49:14.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:13 smithi174 ceph-mon[30177]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:49:14.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:13 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:14.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:13 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:14.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:13 smithi174 ceph-mon[30177]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:49:14.463 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":18,"num_osds":3,"num_up_osds":3,"osd_up_since":1711280953,"num_in_osds":3,"osd_in_since":1711280935,"num_remapped_pgs":0} 2024-03-24T11:49:14.464 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:49:14.740 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:14.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:14 smithi130 ceph-mon[25795]: purged_snaps scrub starts 2024-03-24T11:49:14.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:14 smithi130 ceph-mon[25795]: purged_snaps scrub ok 2024-03-24T11:49:14.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:14 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:14.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:14 smithi130 ceph-mon[25795]: osd.2 [v2:172.21.15.189:6800/627265002,v1:172.21.15.189:6801/627265002] boot 2024-03-24T11:49:14.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:14 smithi130 ceph-mon[25795]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:49:14.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:14 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:14.789 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:14 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/2704082031' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:49:14.806 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:14 smithi189 ceph-mon[30629]: purged_snaps scrub starts 2024-03-24T11:49:14.807 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:14 smithi189 ceph-mon[30629]: purged_snaps scrub ok 2024-03-24T11:49:14.807 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:14 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:14.807 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:14 smithi189 ceph-mon[30629]: osd.2 [v2:172.21.15.189:6800/627265002,v1:172.21.15.189:6801/627265002] boot 2024-03-24T11:49:14.807 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:14 smithi189 ceph-mon[30629]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:49:14.807 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:14 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:14.807 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:14 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/2704082031' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:49:15.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:14 smithi174 ceph-mon[30177]: purged_snaps scrub starts 2024-03-24T11:49:15.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:14 smithi174 ceph-mon[30177]: purged_snaps scrub ok 2024-03-24T11:49:15.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:14 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:15.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:14 smithi174 ceph-mon[30177]: osd.2 [v2:172.21.15.189:6800/627265002,v1:172.21.15.189:6801/627265002] boot 2024-03-24T11:49:15.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:14 smithi174 ceph-mon[30177]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:49:15.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:14 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:15.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:14 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/2704082031' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:49:15.868 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:15 smithi130 ceph-mon[25795]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:49:15.868 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:15 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:15.869 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:15 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:15.869 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:15 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:15.869 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:15 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:15.869 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:15 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:15.869 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:15 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:15.869 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:15 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:16.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:15 smithi174 ceph-mon[30177]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:49:16.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:15 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:16.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:15 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:16.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:15 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:16.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:15 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:16.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:15 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:16.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:15 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:16.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:15 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:16.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:15 smithi189 ceph-mon[30629]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:49:16.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:15 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:16.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:15 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:16.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:15 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:49:16.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:15 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:16.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:15 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:16.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:15 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:16.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:15 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:16.548 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:49:16.548 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":20,"fsid":"e18cf2c8-e9d3-11ee-95cd-87774f69a715","created":"2024-03-24T11:45:53.525144+0000","modified":"2024-03-24T11:49:15.880507+0000","last_up_change":"2024-03-24T11:49:13.643800+0000","last_in_change":"2024-03-24T11:48:55.116392+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:49:15.722605+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d673100d-5f3a-458b-8800-a45d87c7b0c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6803","nonce":2720668797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6805","nonce":2720668797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6809","nonce":2720668797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6807","nonce":2720668797}]},"public_addr":"172.21.15.130:6803/2720668797","cluster_addr":"172.21.15.130:6805/2720668797","heartbeat_back_addr":"172.21.15.130:6809/2720668797","heartbeat_front_addr":"172.21.15.130:6807/2720668797","state":["exists","up"]},{"osd":1,"uuid":"4d623983-bbdf-4791-95e6-f1465b99480c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6801","nonce":3224679846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6803","nonce":3224679846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6807","nonce":3224679846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6805","nonce":3224679846}]},"public_addr":"172.21.15.174:6801/3224679846","cluster_addr":"172.21.15.174:6803/3224679846","heartbeat_back_addr":"172.21.15.174:6807/3224679846","heartbeat_front_addr":"172.21.15.174:6805/3224679846","state":["exists","up"]},{"osd":2,"uuid":"25926a0b-b35b-4d4d-9c4f-ecaf374ee267","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6801","nonce":627265002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6803","nonce":627265002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6807","nonce":627265002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6805","nonce":627265002}]},"public_addr":"172.21.15.189:6801/627265002","cluster_addr":"172.21.15.189:6803/627265002","heartbeat_back_addr":"172.21.15.189:6807/627265002","heartbeat_front_addr":"172.21.15.189:6805/627265002","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:48:23.157485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:48:47.284056+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:49:11.964352+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.130:6801/3779257549":"2024-03-25T11:46:47.390867+0000","172.21.15.130:6800/3779257549":"2024-03-25T11:46:47.390867+0000","172.21.15.130:0/121717340":"2024-03-25T11:46:47.390867+0000","172.21.15.130:0/2771330122":"2024-03-25T11:46:47.390867+0000","172.21.15.130:0/1301989842":"2024-03-25T11:46:21.411624+0000","172.21.15.130:0/3337830437":"2024-03-25T11:46:21.411624+0000","172.21.15.130:6800/1696108947":"2024-03-25T11:46:21.411624+0000","172.21.15.130:6801/1696108947":"2024-03-25T11:46:21.411624+0000","172.21.15.130:0/648618572":"2024-03-25T11:46:21.411624+0000","172.21.15.130:0/2365795870":"2024-03-25T11:46:47.390867+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:49:16.858 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:16 smithi130 ceph-mon[25795]: Detected new or changed devices on smithi189 2024-03-24T11:49:16.858 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:16 smithi130 ceph-mon[25795]: Adjusting osd_memory_target on smithi189 to 17088M 2024-03-24T11:49:16.858 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:16 smithi130 ceph-mon[25795]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:16.858 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:16.858 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:16.858 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:16 smithi130 ceph-mon[25795]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:49:16.858 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:16 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:16.858 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:16 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3324690121' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:49:17.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:16 smithi174 ceph-mon[30177]: Detected new or changed devices on smithi189 2024-03-24T11:49:17.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:16 smithi174 ceph-mon[30177]: Adjusting osd_memory_target on smithi189 to 17088M 2024-03-24T11:49:17.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:16 smithi174 ceph-mon[30177]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:17.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:16 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:17.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:16 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:17.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:16 smithi174 ceph-mon[30177]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:49:17.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:16 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:17.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:16 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3324690121' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:49:17.151 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:49:15.722605+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T11:49:17.152 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T11:49:17.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:16 smithi189 ceph-mon[30629]: Detected new or changed devices on smithi189 2024-03-24T11:49:17.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:16 smithi189 ceph-mon[30629]: Adjusting osd_memory_target on smithi189 to 17088M 2024-03-24T11:49:17.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:16 smithi189 ceph-mon[30629]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:17.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:16 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:17.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:16 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:17.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:16 smithi189 ceph-mon[30629]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:49:17.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:16 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:49:17.197 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:16 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3324690121' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:49:17.390 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:17.493 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:49:17 smithi130 sudo[46617]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-24T11:49:17.493 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:49:17 smithi130 sudo[46617]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:17.493 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:49:17 smithi130 sudo[46617]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:17.493 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:49:17 smithi130 sudo[46617]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:17.788 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:49:17 smithi130 sudo[46674]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-24T11:49:17.788 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:49:17 smithi130 sudo[46674]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:17.788 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:49:17 smithi130 sudo[46674]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:17.788 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:49:17 smithi130 sudo[46674]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:18.015 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:17 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:18.015 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:17 smithi174 ceph-mon[30177]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:49:18.015 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:17 smithi174 ceph-mon[30177]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:18.015 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:17 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:18.015 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:49:17 smithi174 sudo[40402]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-24T11:49:18.016 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:49:17 smithi174 sudo[40402]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:18.016 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:49:17 smithi174 sudo[40402]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:18.016 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:49:17 smithi174 sudo[40402]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:18.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:17 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:18.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:17 smithi189 ceph-mon[30629]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:49:18.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:17 smithi189 ceph-mon[30629]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:18.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:17 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:18.265 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:17 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:49:18.265 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:17 smithi130 ceph-mon[25795]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:49:18.265 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:17 smithi130 ceph-mon[25795]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:18.265 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:17 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:18.344 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:49:18 smithi174 sudo[40405]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-24T11:49:18.344 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:49:18 smithi174 sudo[40405]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:18.344 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:49:18 smithi174 sudo[40405]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:18.344 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:49:18 smithi174 sudo[40405]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:18.533 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:49:18 smithi189 sudo[40980]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-24T11:49:18.533 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:49:18 smithi189 sudo[40980]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:18.533 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:49:18 smithi189 sudo[40980]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:18.533 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:49:18 smithi189 sudo[40980]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:18.946 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:49:18 smithi189 sudo[40983]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-24T11:49:18.946 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:49:18 smithi189 sudo[40983]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:18.946 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:49:18 smithi189 sudo[40983]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:18.946 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:49:18 smithi189 sudo[40983]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:19.063 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:18 smithi130 sudo[46749]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:49:19.064 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:18 smithi130 sudo[46749]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:19.064 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:18 smithi130 sudo[46749]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:19.064 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:18 smithi130 sudo[46749]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:19.278 INFO:teuthology.orchestra.run.smithi130.stdout:pg_num: 1 2024-03-24T11:49:19.333 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 ceph-mon[30629]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:49:19.333 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:19.333 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:19.333 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 ceph-mon[30629]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:49:19.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:19 smithi174 ceph-mon[30177]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:49:19.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:19 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:19.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:19 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:19.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:19 smithi174 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:49:19.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:19 smithi130 sudo[46792]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:49:19.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:19 smithi130 sudo[46792]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:19.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:19 smithi130 sudo[46792]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:19.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:19 smithi130 sudo[46792]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:19.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:19 smithi130 ceph-mon[25795]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:49:19.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:19 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:19.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:19 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:19.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:19 smithi130 ceph-mon[25795]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:49:19.583 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 sudo[40986]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:49:19.584 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 sudo[40986]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:19.584 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 sudo[40986]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:19.584 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 sudo[40986]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:19.584 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 sudo[40989]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:49:19.898 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi130 2024-03-24T11:49:19.898 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi130=foo.a' 2024-03-24T11:49:19.925 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 sudo[40989]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:19.925 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 sudo[40989]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:19.925 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:19 smithi189 sudo[40989]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:20.100 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:19 smithi174 sudo[40408]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:49:20.100 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:19 smithi174 sudo[40408]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:20.100 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:19 smithi174 sudo[40408]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:20.100 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:19 smithi174 sudo[40408]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:20.100 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 sudo[40411]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:49:20.100 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 sudo[40411]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:49:20.100 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 sudo[40411]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:49:20.100 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 sudo[40411]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:49:20.138 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:49:20.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3170751448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:20.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:20.197 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:20.197 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:20 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:20.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:49:20.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:20.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:20.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:20.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-24T11:49:20.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:20.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:49:20.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:20.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:20.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:20.289 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3170751448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:49:20.289 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:20.289 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:49:20.289 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:20.289 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:20.289 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:20.289 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:20 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:20.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:49:20.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:20.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:20.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:20.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-24T11:49:20.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:20.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:49:20.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:20.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:20.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:20.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3170751448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:49:20.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:20.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:49:20.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:20.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:20.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:20.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:20 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:21.263 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:21 smithi189 ceph-mon[30629]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:49:21.263 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:21 smithi189 ceph-mon[30629]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:49:21.263 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:21 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:21.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:21 smithi130 ceph-mon[25795]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:49:21.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:21 smithi130 ceph-mon[25795]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:49:21.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:21 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:21.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:21 smithi174 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:49:21.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:21 smithi174 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:49:21.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:21 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:22.030 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled rgw.foo.a update... 2024-03-24T11:49:22.317 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:22 smithi189 ceph-mon[30629]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:22.318 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:22 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:22.318 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:22 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:22.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:22 smithi130 ceph-mon[25795]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:22.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:22 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:22.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:22 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:22.593 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:22 smithi174 ceph-mon[30177]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:22.593 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:22 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:22.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:22 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:22.666 DEBUG:teuthology.orchestra.run.smithi130:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@rgw.foo.a.service 2024-03-24T11:49:22.669 INFO:tasks.cephadm:Adding prometheus.a on smithi174 2024-03-24T11:49:22.669 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi174=a' 2024-03-24T11:49:22.943 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:49:23.298 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:23 smithi130 ceph-mon[25795]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi130=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:23.298 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:23 smithi130 ceph-mon[25795]: Saving service rgw.foo.a spec with placement smithi130=foo.a;count:1 2024-03-24T11:49:23.298 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:23 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:23.298 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:23 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:23.298 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:23 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:23.298 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:23 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:49:23.299 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:23 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:49:23.299 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:23 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:23.299 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:23 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:23.320 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:23 smithi189 ceph-mon[30629]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi130=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:23.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:23 smithi189 ceph-mon[30629]: Saving service rgw.foo.a spec with placement smithi130=foo.a;count:1 2024-03-24T11:49:23.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:23 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:23.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:23 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:23.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:23 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:23.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:23 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:49:23.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:23 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:49:23.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:23 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:23.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:23 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:23.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:23 smithi174 ceph-mon[30177]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi130=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:23.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:23 smithi174 ceph-mon[30177]: Saving service rgw.foo.a spec with placement smithi130=foo.a;count:1 2024-03-24T11:49:23.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:23 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:23.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:23 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:23.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:23 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:23.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:23 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:49:23.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:23 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:49:23.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:23 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:23.595 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:23 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:24.231 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:24 smithi130 ceph-mon[25795]: Deploying daemon rgw.foo.a on smithi130 2024-03-24T11:49:24.231 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:24 smithi130 ceph-mon[25795]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:24.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:24 smithi189 ceph-mon[30629]: Deploying daemon rgw.foo.a on smithi130 2024-03-24T11:49:24.368 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:24 smithi189 ceph-mon[30629]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:24.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:24 smithi174 ceph-mon[30177]: Deploying daemon rgw.foo.a on smithi130 2024-03-24T11:49:24.594 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:24 smithi174 ceph-mon[30177]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled prometheus update... 2024-03-24T11:49:24.981 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:49:24 smithi130 systemd[1]: Starting Ceph rgw.foo.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:49:25.291 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:49:24 smithi130 podman[47136]: 2024-03-24 11:49:24.883980072 +0000 UTC m=+0.017993281 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:49:25.291 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:49:24 smithi130 podman[47136]: 2024-03-24 11:49:24.988920715 +0000 UTC m=+0.122933975 container create b58dd87162303f1cfe96a59abb07393a98e2dfeed7b148a7cbac2267c5f76075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T11:49:25.291 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:49:25 smithi130 podman[47136]: 2024-03-24 11:49:25.207827475 +0000 UTC m=+0.341840698 container init b58dd87162303f1cfe96a59abb07393a98e2dfeed7b148a7cbac2267c5f76075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T11:49:25.291 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:49:25 smithi130 podman[47136]: 2024-03-24 11:49:25.212919694 +0000 UTC m=+0.346932899 container start b58dd87162303f1cfe96a59abb07393a98e2dfeed7b148a7cbac2267c5f76075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T11:49:25.473 DEBUG:teuthology.orchestra.run.smithi174:prometheus.a> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-24T11:49:25.475 INFO:tasks.cephadm:Adding node-exporter.a on smithi130 2024-03-24T11:49:25.476 INFO:tasks.cephadm:Adding node-exporter.b on smithi174 2024-03-24T11:49:25.476 INFO:tasks.cephadm:Adding node-exporter.c on smithi189 2024-03-24T11:49:25.476 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '3;smithi130=a;smithi174=b;smithi189=c' 2024-03-24T11:49:25.714 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:49:25.787 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:49:25 smithi130 bash[47136]: b58dd87162303f1cfe96a59abb07393a98e2dfeed7b148a7cbac2267c5f76075 2024-03-24T11:49:25.788 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:49:25 smithi130 systemd[1]: Started Ceph rgw.foo.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:49:26.080 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:25 smithi189 ceph-mon[30629]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:26.081 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:25 smithi189 ceph-mon[30629]: Saving service prometheus spec with placement smithi174=a;count:1 2024-03-24T11:49:26.081 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:25 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.081 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:25 smithi189 ceph-mon[30629]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 21 op/s 2024-03-24T11:49:26.081 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:25 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.081 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:25 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.081 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:25 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.081 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:25 smithi189 ceph-mon[30629]: Saving service rgw.foo.a spec with placement smithi130=foo.a;count:1 2024-03-24T11:49:26.081 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:25 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.081 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:25 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.081 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:25 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:26.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:25 smithi174 ceph-mon[30177]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:26.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:25 smithi174 ceph-mon[30177]: Saving service prometheus spec with placement smithi174=a;count:1 2024-03-24T11:49:26.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:25 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:25 smithi174 ceph-mon[30177]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 21 op/s 2024-03-24T11:49:26.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:25 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:25 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:25 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:25 smithi174 ceph-mon[30177]: Saving service rgw.foo.a spec with placement smithi130=foo.a;count:1 2024-03-24T11:49:26.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:25 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:25 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:25 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:26.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:25 smithi130 ceph-mon[25795]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:26.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:25 smithi130 ceph-mon[25795]: Saving service prometheus spec with placement smithi174=a;count:1 2024-03-24T11:49:26.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:25 smithi130 ceph-mon[25795]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 21 op/s 2024-03-24T11:49:26.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:25 smithi130 ceph-mon[25795]: Saving service rgw.foo.a spec with placement smithi130=foo.a;count:1 2024-03-24T11:49:26.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:26.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:25 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:27.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:26 smithi174 ceph-mon[30177]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T11:49:27.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:26 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/74423001' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:49:27.140 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:26 smithi189 ceph-mon[30629]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T11:49:27.141 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:26 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/74423001' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:49:27.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:26 smithi130 ceph-mon[25795]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T11:49:27.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:26 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/74423001' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:49:27.617 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled node-exporter update... 2024-03-24T11:49:28.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:27 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/74423001' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:49:28.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:27 smithi174 ceph-mon[30177]: osdmap e24: 3 total, 3 up, 3 in 2024-03-24T11:49:28.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:27 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:27 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:27 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:28.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:27 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:28.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:27 smithi174 ceph-mon[30177]: Checking dashboard <-> RGW credentials 2024-03-24T11:49:28.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:27 smithi174 ceph-mon[30177]: pgmap v94: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-24T11:49:28.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:27 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:27 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:27 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/74423001' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:49:28.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:27 smithi189 ceph-mon[30629]: osdmap e24: 3 total, 3 up, 3 in 2024-03-24T11:49:28.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:27 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:27 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:27 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:28.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:27 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:28.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:27 smithi189 ceph-mon[30629]: Checking dashboard <-> RGW credentials 2024-03-24T11:49:28.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:27 smithi189 ceph-mon[30629]: pgmap v94: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-24T11:49:28.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:27 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:27 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.243 DEBUG:teuthology.orchestra.run.smithi130:node-exporter.a> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-24T11:49:28.246 DEBUG:teuthology.orchestra.run.smithi174:node-exporter.b> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-24T11:49:28.249 DEBUG:teuthology.orchestra.run.smithi189:node-exporter.c> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@node-exporter.c.service 2024-03-24T11:49:28.251 INFO:tasks.cephadm:Adding alertmanager.a on smithi130 2024-03-24T11:49:28.251 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi130=a' 2024-03-24T11:49:28.271 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:27 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/74423001' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:49:28.271 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:27 smithi130 ceph-mon[25795]: osdmap e24: 3 total, 3 up, 3 in 2024-03-24T11:49:28.272 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:27 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.272 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:27 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.272 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:27 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:49:28.272 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:27 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:49:28.272 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:27 smithi130 ceph-mon[25795]: Checking dashboard <-> RGW credentials 2024-03-24T11:49:28.272 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:27 smithi130 ceph-mon[25795]: pgmap v94: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-24T11:49:28.272 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:27 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.272 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:27 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:28.531 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:49:29.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:28 smithi174 ceph-mon[30177]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi130=a;smithi174=b;smithi189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:29.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:28 smithi174 ceph-mon[30177]: Saving service node-exporter spec with placement smithi130=a;smithi174=b;smithi189=c;count:3 2024-03-24T11:49:29.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:28 smithi174 ceph-mon[30177]: osdmap e25: 3 total, 3 up, 3 in 2024-03-24T11:49:29.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:28 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:49:29.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:28 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:49:29.036 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:28 smithi189 ceph-mon[30629]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi130=a;smithi174=b;smithi189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:29.036 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:28 smithi189 ceph-mon[30629]: Saving service node-exporter spec with placement smithi130=a;smithi174=b;smithi189=c;count:3 2024-03-24T11:49:29.036 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:28 smithi189 ceph-mon[30629]: osdmap e25: 3 total, 3 up, 3 in 2024-03-24T11:49:29.036 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:28 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:49:29.036 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:28 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:49:29.287 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:28 smithi130 ceph-mon[25795]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi130=a;smithi174=b;smithi189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:29.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:28 smithi130 ceph-mon[25795]: Saving service node-exporter spec with placement smithi130=a;smithi174=b;smithi189=c;count:3 2024-03-24T11:49:29.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:28 smithi130 ceph-mon[25795]: osdmap e25: 3 total, 3 up, 3 in 2024-03-24T11:49:29.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:28 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:49:29.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:28 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:49:30.067 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:29 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:49:30.067 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:29 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:49:30.067 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:29 smithi189 ceph-mon[30629]: osdmap e26: 3 total, 3 up, 3 in 2024-03-24T11:49:30.067 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:29 smithi189 ceph-mon[30629]: pgmap v97: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:30.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:29 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:49:30.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:29 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:49:30.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:29 smithi174 ceph-mon[30177]: osdmap e26: 3 total, 3 up, 3 in 2024-03-24T11:49:30.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:29 smithi174 ceph-mon[30177]: pgmap v97: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:30.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:29 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:49:30.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:29 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:49:30.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:29 smithi130 ceph-mon[25795]: osdmap e26: 3 total, 3 up, 3 in 2024-03-24T11:49:30.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:29 smithi130 ceph-mon[25795]: pgmap v97: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:30.514 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled alertmanager update... 2024-03-24T11:49:31.078 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:30 smithi189 ceph-mon[30629]: osdmap e27: 3 total, 3 up, 3 in 2024-03-24T11:49:31.079 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:30 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:49:31.079 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:30 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:49:31.079 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:30 smithi189 ceph-mon[30629]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi130=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:31.079 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:30 smithi189 ceph-mon[30629]: Saving service alertmanager spec with placement smithi130=a;count:1 2024-03-24T11:49:31.079 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:30 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:31.104 DEBUG:teuthology.orchestra.run.smithi130:alertmanager.a> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-24T11:49:31.106 INFO:tasks.cephadm:Adding grafana.a on smithi174 2024-03-24T11:49:31.107 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi174=a' 2024-03-24T11:49:31.128 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:30 smithi130 ceph-mon[25795]: osdmap e27: 3 total, 3 up, 3 in 2024-03-24T11:49:31.128 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:30 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:49:31.129 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:30 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:49:31.129 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:30 smithi130 ceph-mon[25795]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi130=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:31.129 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:30 smithi130 ceph-mon[25795]: Saving service alertmanager spec with placement smithi130=a;count:1 2024-03-24T11:49:31.129 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:30 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:31.343 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:49:31.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:30 smithi174 ceph-mon[30177]: osdmap e27: 3 total, 3 up, 3 in 2024-03-24T11:49:31.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:30 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:49:31.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:30 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:49:31.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:30 smithi174 ceph-mon[30177]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi130=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:31.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:30 smithi174 ceph-mon[30177]: Saving service alertmanager spec with placement smithi130=a;count:1 2024-03-24T11:49:31.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:30 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:32.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:31 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:49:32.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:31 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:49:32.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:31 smithi189 ceph-mon[30629]: osdmap e28: 3 total, 3 up, 3 in 2024-03-24T11:49:32.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:31 smithi189 ceph-mon[30629]: pgmap v100: 97 pgs: 11 creating+peering, 21 unknown, 65 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T11:49:32.287 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:31 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:49:32.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:31 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:49:32.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:31 smithi130 ceph-mon[25795]: osdmap e28: 3 total, 3 up, 3 in 2024-03-24T11:49:32.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:31 smithi130 ceph-mon[25795]: pgmap v100: 97 pgs: 11 creating+peering, 21 unknown, 65 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T11:49:32.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:31 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:49:32.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:31 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:49:32.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:31 smithi174 ceph-mon[30177]: osdmap e28: 3 total, 3 up, 3 in 2024-03-24T11:49:32.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:31 smithi174 ceph-mon[30177]: pgmap v100: 97 pgs: 11 creating+peering, 21 unknown, 65 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T11:49:33.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:32 smithi189 ceph-mon[30629]: osdmap e29: 3 total, 3 up, 3 in 2024-03-24T11:49:33.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:32 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:49:33.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:32 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:49:33.287 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:32 smithi130 ceph-mon[25795]: osdmap e29: 3 total, 3 up, 3 in 2024-03-24T11:49:33.287 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:32 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:49:33.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:32 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:49:33.288 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled grafana update... 2024-03-24T11:49:33.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:32 smithi174 ceph-mon[30177]: osdmap e29: 3 total, 3 up, 3 in 2024-03-24T11:49:33.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:32 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:49:33.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:32 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:49:33.920 DEBUG:teuthology.orchestra.run.smithi174:grafana.a> sudo journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@grafana.a.service 2024-03-24T11:49:33.923 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T11:49:33.923 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:49:34.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:49:34.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:49:34.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: osdmap e30: 3 total, 3 up, 3 in 2024-03-24T11:49:34.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:49:34.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:49:34.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:34.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: Saving service grafana spec with placement smithi174=a;count:1 2024-03-24T11:49:34.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:34.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: pgmap v103: 129 pgs: 11 creating+peering, 53 unknown, 65 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T11:49:34.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:49:34.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:49:34.039 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:33 smithi130 ceph-mon[25795]: osdmap e31: 3 total, 3 up, 3 in 2024-03-24T11:49:34.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:49:34.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:49:34.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: osdmap e30: 3 total, 3 up, 3 in 2024-03-24T11:49:34.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:49:34.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:49:34.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:34.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: Saving service grafana spec with placement smithi174=a;count:1 2024-03-24T11:49:34.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:34.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: pgmap v103: 129 pgs: 11 creating+peering, 53 unknown, 65 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T11:49:34.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:49:34.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:49:34.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:33 smithi189 ceph-mon[30629]: osdmap e31: 3 total, 3 up, 3 in 2024-03-24T11:49:34.217 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:34.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:49:34.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:49:34.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: osdmap e30: 3 total, 3 up, 3 in 2024-03-24T11:49:34.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:49:34.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:49:34.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:49:34.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: Saving service grafana spec with placement smithi174=a;count:1 2024-03-24T11:49:34.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:34.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: pgmap v103: 129 pgs: 11 creating+peering, 53 unknown, 65 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T11:49:34.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3638680336' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:49:34.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3507205297' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:49:34.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:33 smithi174 ceph-mon[30177]: osdmap e31: 3 total, 3 up, 3 in 2024-03-24T11:49:35.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:35 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:35.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:35 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:35.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:35 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:35.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:35 smithi130 ceph-mon[25795]: Deploying daemon prometheus.a on smithi174 2024-03-24T11:49:35.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:35 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:35.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:35 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:35.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:35 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:35.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:35 smithi174 ceph-mon[30177]: Deploying daemon prometheus.a on smithi174 2024-03-24T11:49:35.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:35 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:35.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:35 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:35.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:35 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:35.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:35 smithi189 ceph-mon[30629]: Deploying daemon prometheus.a on smithi174 2024-03-24T11:49:36.129 INFO:teuthology.orchestra.run.smithi130.stdout:[client.0] 2024-03-24T11:49:36.129 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQBQEwBmpjKgBxAAAtOWTjzNGKuH7T5MKiR6+A== 2024-03-24T11:49:36.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:36 smithi174 ceph-mon[30177]: pgmap v105: 129 pgs: 11 creating+peering, 10 unknown, 108 active+clean; 581 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 65 KiB/s rd, 5.3 KiB/s wr, 157 op/s 2024-03-24T11:49:36.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:36 smithi189 ceph-mon[30629]: pgmap v105: 129 pgs: 11 creating+peering, 10 unknown, 108 active+clean; 581 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 65 KiB/s rd, 5.3 KiB/s wr, 157 op/s 2024-03-24T11:49:36.506 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:36 smithi130 ceph-mon[25795]: pgmap v105: 129 pgs: 11 creating+peering, 10 unknown, 108 active+clean; 581 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 65 KiB/s rd, 5.3 KiB/s wr, 157 op/s 2024-03-24T11:49:36.809 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:49:36.809 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T11:49:36.810 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T11:49:36.852 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:49:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:49:37.211 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:37 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/69018836' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:37.212 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:37 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/69018836' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:49:37.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:37 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/69018836' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:37.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:37 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/69018836' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:49:37.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:37 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/69018836' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:37.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:37 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/69018836' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:49:38.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:38 smithi174 ceph-mon[30177]: pgmap v106: 129 pgs: 129 active+clean; 582 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 77 KiB/s rd, 6.0 KiB/s wr, 187 op/s 2024-03-24T11:49:38.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:38 smithi189 ceph-mon[30629]: pgmap v106: 129 pgs: 129 active+clean; 582 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 77 KiB/s rd, 6.0 KiB/s wr, 187 op/s 2024-03-24T11:49:38.538 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:38 smithi130 ceph-mon[25795]: pgmap v106: 129 pgs: 129 active+clean; 582 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 77 KiB/s rd, 6.0 KiB/s wr, 187 op/s 2024-03-24T11:49:39.755 INFO:teuthology.orchestra.run.smithi174.stdout:[client.1] 2024-03-24T11:49:39.755 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQBTEwBmWs+rLBAAsBgO+dtRZXGxGlLRJHoAYw== 2024-03-24T11:49:40.713 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:40 smithi174 ceph-mon[30177]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 62 KiB/s rd, 4.8 KiB/s wr, 149 op/s 2024-03-24T11:49:40.713 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:40 smithi174 ceph-mon[30177]: from='client.? 172.21.15.174:0/59591322' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:40.713 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:40 smithi174 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:40.713 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:40 smithi174 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:49:40.738 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-24T11:49:40.739 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-24T11:49:40.739 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T11:49:40.781 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:49:40.787 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:40 smithi130 ceph-mon[25795]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 62 KiB/s rd, 4.8 KiB/s wr, 149 op/s 2024-03-24T11:49:40.787 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:40 smithi130 ceph-mon[25795]: from='client.? 172.21.15.174:0/59591322' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:40.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:40 smithi130 ceph-mon[25795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:40.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:40 smithi130 ceph-mon[25795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:49:40.811 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:40 smithi189 ceph-mon[30629]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 62 KiB/s rd, 4.8 KiB/s wr, 149 op/s 2024-03-24T11:49:40.811 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:40 smithi189 ceph-mon[30629]: from='client.? 172.21.15.174:0/59591322' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:40.811 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:40 smithi189 ceph-mon[30629]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:40.811 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:40 smithi189 ceph-mon[30629]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:49:41.029 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:49:42.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:42 smithi130 ceph-mon[25795]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 131 op/s 2024-03-24T11:49:42.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:42 smithi174 ceph-mon[30177]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 131 op/s 2024-03-24T11:49:42.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:42 smithi189 ceph-mon[30629]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 131 op/s 2024-03-24T11:49:42.971 INFO:teuthology.orchestra.run.smithi189.stdout:[client.2] 2024-03-24T11:49:42.971 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBWEwBmAWaTORAA+oC3wS8muwgfvCIka/qrlg== 2024-03-24T11:49:43.595 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:43 smithi189 ceph-mon[30629]: from='client.? 172.21.15.189:0/695946972' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:43.596 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:43 smithi189 ceph-mon[30629]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:43.596 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:43 smithi189 ceph-mon[30629]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:49:43.621 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:49:43.621 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-24T11:49:43.621 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-24T11:49:43.704 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T11:49:43.704 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T11:49:43.704 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T11:49:43.732 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:43 smithi130 ceph-mon[25795]: from='client.? 172.21.15.189:0/695946972' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:43.732 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:43 smithi130 ceph-mon[25795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:43.732 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:43 smithi130 ceph-mon[25795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:49:43.843 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:43 smithi174 ceph-mon[30177]: from='client.? 172.21.15.189:0/695946972' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:43.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:43 smithi174 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:49:43.844 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:43 smithi174 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:49:43.950 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:44.762 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:44 smithi130 ceph-mon[25795]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.6 KiB/s wr, 112 op/s 2024-03-24T11:49:44.843 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:44 smithi174 ceph-mon[30177]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.6 KiB/s wr, 112 op/s 2024-03-24T11:49:44.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:44 smithi189 ceph-mon[30629]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.6 KiB/s wr, 112 op/s 2024-03-24T11:49:45.770 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:49:46.420 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":78955633},{"type":"v1","addr":"172.21.15.130:6801","nonce":78955633}]},"active_addr":"172.21.15.130:6801/78955633","active_change":"2024-03-24T11:46:47.391019+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.130:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.130:0","nonce":2463965632}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.130:0","nonce":2327817451}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.130:0","nonce":3869115851}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.130:0","nonce":87466740}]}]} 2024-03-24T11:49:46.425 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T11:49:46.426 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T11:49:46.426 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:49:46.661 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:46.736 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:46 smithi174 ceph-mon[30177]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 40 KiB/s rd, 3.1 KiB/s wr, 97 op/s 2024-03-24T11:49:46.737 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:46 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/183010243' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:49:46.782 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:46 smithi130 ceph-mon[25795]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 40 KiB/s rd, 3.1 KiB/s wr, 97 op/s 2024-03-24T11:49:46.783 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:46 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/183010243' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:49:46.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:46 smithi189 ceph-mon[30629]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 40 KiB/s rd, 3.1 KiB/s wr, 97 op/s 2024-03-24T11:49:46.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:46 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/183010243' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:49:48.464 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:49:48.464 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":31,"fsid":"e18cf2c8-e9d3-11ee-95cd-87774f69a715","created":"2024-03-24T11:45:53.525144+0000","modified":"2024-03-24T11:49:33.868549+0000","last_up_change":"2024-03-24T11:49:13.643800+0000","last_in_change":"2024-03-24T11:48:55.116392+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:49:15.722605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-24T11:49:25.487063+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:49:27.140461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:49:29.087831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:49:31.156451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d673100d-5f3a-458b-8800-a45d87c7b0c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6803","nonce":2720668797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6805","nonce":2720668797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6809","nonce":2720668797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6807","nonce":2720668797}]},"public_addr":"172.21.15.130:6803/2720668797","cluster_addr":"172.21.15.130:6805/2720668797","heartbeat_back_addr":"172.21.15.130:6809/2720668797","heartbeat_front_addr":"172.21.15.130:6807/2720668797","state":["exists","up"]},{"osd":1,"uuid":"4d623983-bbdf-4791-95e6-f1465b99480c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6801","nonce":3224679846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6803","nonce":3224679846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6807","nonce":3224679846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6805","nonce":3224679846}]},"public_addr":"172.21.15.174:6801/3224679846","cluster_addr":"172.21.15.174:6803/3224679846","heartbeat_back_addr":"172.21.15.174:6807/3224679846","heartbeat_front_addr":"172.21.15.174:6805/3224679846","state":["exists","up"]},{"osd":2,"uuid":"25926a0b-b35b-4d4d-9c4f-ecaf374ee267","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6801","nonce":627265002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6803","nonce":627265002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6807","nonce":627265002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6805","nonce":627265002}]},"public_addr":"172.21.15.189:6801/627265002","cluster_addr":"172.21.15.189:6803/627265002","heartbeat_back_addr":"172.21.15.189:6807/627265002","heartbeat_front_addr":"172.21.15.189:6805/627265002","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:48:23.157485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:48:47.284056+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:49:11.964352+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.130:6801/3779257549":"2024-03-25T11:46:47.390867+0000","172.21.15.130:6800/3779257549":"2024-03-25T11:46:47.390867+0000","172.21.15.130:0/121717340":"2024-03-25T11:46:47.390867+0000","172.21.15.130:0/2771330122":"2024-03-25T11:46:47.390867+0000","172.21.15.130:0/1301989842":"2024-03-25T11:46:21.411624+0000","172.21.15.130:0/3337830437":"2024-03-25T11:46:21.411624+0000","172.21.15.130:6800/1696108947":"2024-03-25T11:46:21.411624+0000","172.21.15.130:6801/1696108947":"2024-03-25T11:46:21.411624+0000","172.21.15.130:0/648618572":"2024-03-25T11:46:21.411624+0000","172.21.15.130:0/2365795870":"2024-03-25T11:46:47.390867+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:49:48.489 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 systemd[1]: Starting Ceph prometheus.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:49:48.764 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:48 smithi130 ceph-mon[25795]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 14 KiB/s rd, 1023 B/s wr, 33 op/s 2024-03-24T11:49:48.764 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:48 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/3053362816' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:49:48.788 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-mon[30177]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 14 KiB/s rd, 1023 B/s wr, 33 op/s 2024-03-24T11:49:48.788 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/3053362816' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:49:48.788 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 podman[41327]: 2024-03-24 11:49:48.487854214 +0000 UTC m=+0.019414717 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-24T11:49:48.788 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 podman[41327]: 2024-03-24 11:49:48.595224301 +0000 UTC m=+0.126784789 container create 6e5961b3423f7c92c9fe9b2e61fbda60d4cafde60a7eef1a4874e744dd036130 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:49:48.788 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 podman[41327]: 2024-03-24 11:49:48.788647649 +0000 UTC m=+0.320208137 container init 6e5961b3423f7c92c9fe9b2e61fbda60d4cafde60a7eef1a4874e744dd036130 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:49:48.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:48 smithi189 ceph-mon[30629]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 14 KiB/s rd, 1023 B/s wr, 33 op/s 2024-03-24T11:49:48.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:48 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/3053362816' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:49:49.094 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T11:49:49.095 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:49:49.095 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 podman[41327]: 2024-03-24 11:49:48.794126231 +0000 UTC m=+0.325686718 container start 6e5961b3423f7c92c9fe9b2e61fbda60d4cafde60a7eef1a4874e744dd036130 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:49:49.095 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.817Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T11:49:49.095 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.817Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.817Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi174 (none))" 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.817Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.817Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.819Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.819Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.820Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.820Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.821Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.821Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.39µs 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.821Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.822Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.822Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.853µs wal_replay_duration=343.906µs wbl_replay_duration=126ns total_replay_duration=396.974µs 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.823Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.823Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T11:49:49.096 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.823Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:49:49.097 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.840Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.896937ms db_storage=813ns remote_storage=1.329µs web_handler=433ns query_engine=526ns scrape=688.703µs scrape_sd=121.849µs notify=1.232µs notify_sd=6.956µs rules=16.894195ms tracing=5.253µs 2024-03-24T11:49:49.097 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.841Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T11:49:49.097 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:49:48.841Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T11:49:49.097 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 bash[41327]: 6e5961b3423f7c92c9fe9b2e61fbda60d4cafde60a7eef1a4874e744dd036130 2024-03-24T11:49:49.097 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:49:48 smithi174 systemd[1]: Started Ceph prometheus.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:49:49.328 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:49.998 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:49.998 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:49.998 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:49.998 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:49 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:49:49.998 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:49 smithi130 ceph-mon[25795]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:50.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:50.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:50.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:50.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:49 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:49:50.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:49 smithi189 ceph-mon[30629]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:50.196 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:50 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: ignoring --setuser ceph since I am not root 2024-03-24T11:49:50.196 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:50 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: ignoring --setgroup ceph since I am not root 2024-03-24T11:49:50.196 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:50 smithi189 ceph-mgr[32763]: -- 172.21.15.189:0/2321441980 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5592bbb2e1a0 con 0x5592bbb05000 2024-03-24T11:49:50.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:50 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: ignoring --setuser ceph since I am not root 2024-03-24T11:49:50.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:50 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: ignoring --setgroup ceph since I am not root 2024-03-24T11:49:50.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:50.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:50.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' 2024-03-24T11:49:50.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:49 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:49:50.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:49 smithi174 ceph-mon[30177]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:49:50.344 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:50 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: ignoring --setuser ceph since I am not root 2024-03-24T11:49:50.344 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:50 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: ignoring --setgroup ceph since I am not root 2024-03-24T11:49:50.344 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:50 smithi174 ceph-mgr[31668]: -- 172.21.15.174:0/2104246084 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cd638b21a0 con 0x55cd63888800 2024-03-24T11:49:50.705 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:50 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:50.541+0000 7f4b7c79d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:49:50.705 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:50 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:50.637+0000 7f4b7c79d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:49:50.843 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:50 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:50.527+0000 7ff517802200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:49:50.844 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:50 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:50.619+0000 7ff517802200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:49:50.945 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:50 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:50.532+0000 7f7010b38200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:49:50.945 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:50 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:50.625+0000 7f7010b38200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:49:51.229 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:49:51.229 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":31,"fsid":"e18cf2c8-e9d3-11ee-95cd-87774f69a715","created":"2024-03-24T11:45:53.525144+0000","modified":"2024-03-24T11:49:33.868549+0000","last_up_change":"2024-03-24T11:49:13.643800+0000","last_in_change":"2024-03-24T11:48:55.116392+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:49:15.722605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-24T11:49:25.487063+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:49:27.140461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:49:29.087831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:49:31.156451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d673100d-5f3a-458b-8800-a45d87c7b0c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6803","nonce":2720668797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6805","nonce":2720668797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6809","nonce":2720668797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2720668797},{"type":"v1","addr":"172.21.15.130:6807","nonce":2720668797}]},"public_addr":"172.21.15.130:6803/2720668797","cluster_addr":"172.21.15.130:6805/2720668797","heartbeat_back_addr":"172.21.15.130:6809/2720668797","heartbeat_front_addr":"172.21.15.130:6807/2720668797","state":["exists","up"]},{"osd":1,"uuid":"4d623983-bbdf-4791-95e6-f1465b99480c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6801","nonce":3224679846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6803","nonce":3224679846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6807","nonce":3224679846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3224679846},{"type":"v1","addr":"172.21.15.174:6805","nonce":3224679846}]},"public_addr":"172.21.15.174:6801/3224679846","cluster_addr":"172.21.15.174:6803/3224679846","heartbeat_back_addr":"172.21.15.174:6807/3224679846","heartbeat_front_addr":"172.21.15.174:6805/3224679846","state":["exists","up"]},{"osd":2,"uuid":"25926a0b-b35b-4d4d-9c4f-ecaf374ee267","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6801","nonce":627265002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6803","nonce":627265002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6807","nonce":627265002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":627265002},{"type":"v1","addr":"172.21.15.189:6805","nonce":627265002}]},"public_addr":"172.21.15.189:6801/627265002","cluster_addr":"172.21.15.189:6803/627265002","heartbeat_back_addr":"172.21.15.189:6807/627265002","heartbeat_front_addr":"172.21.15.189:6805/627265002","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:48:23.157485+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:48:47.284056+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:49:11.964352+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.130:6801/3779257549":"2024-03-25T11:46:47.390867+0000","172.21.15.130:6800/3779257549":"2024-03-25T11:46:47.390867+0000","172.21.15.130:0/121717340":"2024-03-25T11:46:47.390867+0000","172.21.15.130:0/2771330122":"2024-03-25T11:46:47.390867+0000","172.21.15.130:0/1301989842":"2024-03-25T11:46:21.411624+0000","172.21.15.130:0/3337830437":"2024-03-25T11:46:21.411624+0000","172.21.15.130:6800/1696108947":"2024-03-25T11:46:21.411624+0000","172.21.15.130:6801/1696108947":"2024-03-25T11:46:21.411624+0000","172.21.15.130:0/648618572":"2024-03-25T11:46:21.411624+0000","172.21.15.130:0/2365795870":"2024-03-25T11:46:47.390867+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:49:51.229 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:50 smithi130 ceph-mon[25795]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:49:51.229 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:50 smithi130 ceph-mon[25795]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-24T11:49:51.246 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:50 smithi174 ceph-mon[30177]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:49:51.246 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:50 smithi174 ceph-mon[30177]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-24T11:49:51.268 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:50 smithi189 ceph-mon[30629]: from='mgr.14152 172.21.15.130:0/2511284323' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:49:51.268 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:50 smithi189 ceph-mon[30629]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-24T11:49:51.537 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:51 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:51.306+0000 7f4b7c79d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:49:51.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:51 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:51.400+0000 7f4b7c79d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:49:51.570 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:51 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:51.245+0000 7ff517802200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:49:51.570 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:51 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:51.337+0000 7ff517802200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:49:51.570 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:51 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:51.475+0000 7ff517802200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:49:51.570 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:51 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:51.569+0000 7ff517802200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:49:51.599 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:51 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:51.267+0000 7f7010b38200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:49:51.600 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:51 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:51.360+0000 7f7010b38200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:49:51.600 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:51 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:51.502+0000 7f7010b38200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:49:51.600 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:51 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:51.599+0000 7f7010b38200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:49:51.836 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:51 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:51.543+0000 7f4b7c79d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:49:51.836 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:51 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:51.641+0000 7f4b7c79d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:49:51.836 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:51 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:51.724+0000 7f4b7c79d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:49:51.843 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:51 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:51.651+0000 7ff517802200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:49:51.864 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T11:49:51.864 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T11:49:51.864 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T11:49:51.945 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:51 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:51.683+0000 7f7010b38200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:49:52.146 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:52.146 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:52.148 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:52.149 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:51 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/2885461118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:49:52.149 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:52 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:52.000+0000 7f4b7c79d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:49:52.149 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:52 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:52.148+0000 7f4b7c79d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:49:52.194 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:51 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/2885461118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:49:52.195 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:51 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:51.910+0000 7ff517802200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:49:52.195 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:52 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:52.018+0000 7ff517802200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:49:52.195 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:52 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:52.101+0000 7ff517802200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:49:52.195 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:52 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:52.193+0000 7ff517802200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:49:52.243 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:51 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/2885461118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:49:52.243 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:51 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:51.952+0000 7f7010b38200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:49:52.243 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:52 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:52.063+0000 7f7010b38200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:49:52.243 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:52 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:52.148+0000 7f7010b38200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:49:52.243 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:52 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:52.242+0000 7f7010b38200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:49:52.405 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:52 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:52.247+0000 7f4b7c79d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:49:52.405 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:52 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:52.343+0000 7f4b7c79d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:49:52.593 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:52 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:52.367+0000 7ff517802200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:49:52.657 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:52 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:52.527+0000 7f4b7c79d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:49:52.695 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:52 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:52.419+0000 7f7010b38200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:49:53.593 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:53 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:53.312+0000 7ff517802200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:49:53.594 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:53 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:53.440+0000 7ff517802200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:49:53.695 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:53 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:53.383+0000 7f7010b38200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:49:53.695 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:53 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:53.513+0000 7f7010b38200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:49:53.787 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:53 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:53.504+0000 7f4b7c79d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:49:53.788 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:53 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:53.641+0000 7f4b7c79d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:49:54.048 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:53 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:53.770+0000 7ff517802200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:49:54.048 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:53 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:53.857+0000 7ff517802200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:49:54.048 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:54 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:54.047+0000 7ff517802200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:49:54.137 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:53 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:53.849+0000 7f7010b38200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:49:54.137 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:53 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:53.940+0000 7f7010b38200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:49:54.137 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:54 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:54.136+0000 7f7010b38200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:49:54.286 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:53 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:53.980+0000 7f4b7c79d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:49:54.286 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:54 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:54.071+0000 7f4b7c79d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:49:54.343 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:54 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:54.226+0000 7ff517802200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:49:54.445 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:54 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:54.322+0000 7f7010b38200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:49:54.537 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:54 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:54.285+0000 7f4b7c79d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:49:54.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:54 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:54.485+0000 7f4b7c79d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:49:54.767 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:54 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:54.505+0000 7ff517802200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:49:54.767 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:54 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:54.766+0000 7ff517802200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:49:54.873 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:54 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:54.609+0000 7f7010b38200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:49:54.873 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:54 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:54.872+0000 7f7010b38200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:49:55.078 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:54 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:54.806+0000 7f4b7c79d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:49:55.093 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:55 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:55.004+0000 7ff517802200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:49:55.195 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:55 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:55.107+0000 7f7010b38200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:49:55.330 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:55 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:55.077+0000 7f4b7c79d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:49:55.330 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:55 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:55.323+0000 7f4b7c79d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:49:55.913 INFO:teuthology.orchestra.run.smithi130.stdout:34359738388 2024-03-24T11:49:55.914 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T11:49:56.093 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:55 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:55.801+0000 7ff517802200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:49:56.195 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:55 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:55.908+0000 7f7010b38200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:49:56.495 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:56 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:56.179+0000 7f4b7c79d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:49:56.689 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:56.374+0000 7ff517802200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:49:56.689 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:56.474+0000 7ff517802200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:49:56.689 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:56.688+0000 7ff517802200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:49:56.763 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:56.482+0000 7f7010b38200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:49:56.763 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:56.580+0000 7f7010b38200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:49:57.017 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:56 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:56.773+0000 7f4b7c79d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:49:57.017 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:56 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:56.874+0000 7f4b7c79d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:49:57.017 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:56 smithi130 ceph-mon[25795]: Standby manager daemon x restarted 2024-03-24T11:49:57.018 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:56 smithi130 ceph-mon[25795]: Standby manager daemon x started 2024-03-24T11:49:57.018 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:56 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:49:57.018 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:56 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:49:57.018 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:56 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:49:57.018 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:56 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:49:57.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-mon[30177]: Standby manager daemon x restarted 2024-03-24T11:49:57.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-mon[30177]: Standby manager daemon x started 2024-03-24T11:49:57.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:49:57.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:49:57.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:49:57.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:49:57.094 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: [24/Mar/2024:11:49:56] ENGINE Bus STARTING 2024-03-24T11:49:57.094 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: CherryPy Checker: 2024-03-24T11:49:57.094 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: The Application mounted at '' has an empty config. 2024-03-24T11:49:57.094 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: 2024-03-24T11:49:57.094 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: [24/Mar/2024:11:49:56] ENGINE Serving on http://:::9283 2024-03-24T11:49:57.094 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:49:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x[31645]: [24/Mar/2024:11:49:56] ENGINE Bus STARTED 2024-03-24T11:49:57.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-mon[30629]: Standby manager daemon x restarted 2024-03-24T11:49:57.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-mon[30629]: Standby manager daemon x started 2024-03-24T11:49:57.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:49:57.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:49:57.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:49:57.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.174:0/1599499550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:49:57.196 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:56.800+0000 7f7010b38200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:49:57.197 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: [24/Mar/2024:11:49:56] ENGINE Bus STARTING 2024-03-24T11:49:57.197 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: CherryPy Checker: 2024-03-24T11:49:57.197 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: The Application mounted at '' has an empty config. 2024-03-24T11:49:57.197 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: 2024-03-24T11:49:57.197 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: [24/Mar/2024:11:49:56] ENGINE Serving on http://:::9283 2024-03-24T11:49:57.197 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:49:56 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z[32735]: [24/Mar/2024:11:49:56] ENGINE Bus STARTED 2024-03-24T11:49:57.273 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:57.129+0000 7f4b7c79d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:49:57.274 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:49:57] ENGINE Bus STARTING 2024-03-24T11:49:57.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: CherryPy Checker: 2024-03-24T11:49:57.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: The Application mounted at '' has an empty config. 2024-03-24T11:49:57.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:49:57.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:49:57] ENGINE Serving on http://:::9283 2024-03-24T11:49:57.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:49:57] ENGINE Bus STARTED 2024-03-24T11:49:57.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-24T11:49:57.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: Standby manager daemon z restarted 2024-03-24T11:49:57.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: Standby manager daemon z started 2024-03-24T11:49:57.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:49:57.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:49:57.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:49:57.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:49:57.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: Active manager daemon y restarted 2024-03-24T11:49:57.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: Activating manager daemon y 2024-03-24T11:49:57.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: osdmap e32: 3 total, 3 up, 3 in 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: mgrmap e19: y(active, starting, since 0.0130431s), standbys: x, z 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: Manager daemon y is now available 2024-03-24T11:49:57.875 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:57.876 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:57.876 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:49:57.876 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:57.876 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:49:57.876 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:57 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:49:57.901 INFO:teuthology.orchestra.run.smithi130.stdout:55834574864 2024-03-24T11:49:57.902 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T11:49:57.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: Standby manager daemon z restarted 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: Standby manager daemon z started 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: Active manager daemon y restarted 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: Activating manager daemon y 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: osdmap e32: 3 total, 3 up, 3 in 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: mgrmap e19: y(active, starting, since 0.0130431s), standbys: x, z 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:57.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: Manager daemon y is now available 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:57.947 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:49:57.948 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:57.948 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:49:57.948 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:57 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: Standby manager daemon z restarted 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: Standby manager daemon z started 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.? 172.21.15.189:0/2025758562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: Active manager daemon y restarted 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: Activating manager daemon y 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: osdmap e32: 3 total, 3 up, 3 in 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: mgrmap e19: y(active, starting, since 0.0130431s), standbys: x, z 2024-03-24T11:49:58.095 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: Manager daemon y is now available 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:49:58.096 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:49:58.097 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:49:58.097 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:49:58.097 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:57 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:49:58.110 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:58.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:58 smithi130 ceph-mon[25795]: [24/Mar/2024:11:49:57] ENGINE Bus STARTING 2024-03-24T11:49:58.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:58 smithi130 ceph-mon[25795]: [24/Mar/2024:11:49:57] ENGINE Serving on https://172.21.15.130:7150 2024-03-24T11:49:58.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:58 smithi130 ceph-mon[25795]: [24/Mar/2024:11:49:57] ENGINE Serving on http://172.21.15.130:8765 2024-03-24T11:49:58.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:58 smithi130 ceph-mon[25795]: [24/Mar/2024:11:49:57] ENGINE Bus STARTED 2024-03-24T11:49:58.788 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:58 smithi130 ceph-mon[25795]: mgrmap e20: y(active, since 1.01698s), standbys: x, z 2024-03-24T11:49:58.814 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:49:59.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:58 smithi174 ceph-mon[30177]: [24/Mar/2024:11:49:57] ENGINE Bus STARTING 2024-03-24T11:49:59.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:58 smithi174 ceph-mon[30177]: [24/Mar/2024:11:49:57] ENGINE Serving on https://172.21.15.130:7150 2024-03-24T11:49:59.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:58 smithi174 ceph-mon[30177]: [24/Mar/2024:11:49:57] ENGINE Serving on http://172.21.15.130:8765 2024-03-24T11:49:59.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:58 smithi174 ceph-mon[30177]: [24/Mar/2024:11:49:57] ENGINE Bus STARTED 2024-03-24T11:49:59.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:58 smithi174 ceph-mon[30177]: mgrmap e20: y(active, since 1.01698s), standbys: x, z 2024-03-24T11:49:59.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:58 smithi189 ceph-mon[30629]: [24/Mar/2024:11:49:57] ENGINE Bus STARTING 2024-03-24T11:49:59.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:58 smithi189 ceph-mon[30629]: [24/Mar/2024:11:49:57] ENGINE Serving on https://172.21.15.130:7150 2024-03-24T11:49:59.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:58 smithi189 ceph-mon[30629]: [24/Mar/2024:11:49:57] ENGINE Serving on http://172.21.15.130:8765 2024-03-24T11:49:59.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:58 smithi189 ceph-mon[30629]: [24/Mar/2024:11:49:57] ENGINE Bus STARTED 2024-03-24T11:49:59.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:58 smithi189 ceph-mon[30629]: mgrmap e20: y(active, since 1.01698s), standbys: x, z 2024-03-24T11:49:59.837 INFO:teuthology.orchestra.run.smithi130.stdout:77309411338 2024-03-24T11:49:59.838 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T11:50:00.180 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:59 smithi130 ceph-mon[25795]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:00.180 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:59 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.180 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:59 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.180 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:59 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.180 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:49:59 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:59 smithi189 ceph-mon[30629]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:00.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:59 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:59 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:59 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:49:59 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.289 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:59 smithi174 ceph-mon[30177]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:00.289 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:59 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.289 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:59 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.289 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:59 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.289 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:49:59 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:00.969 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:01.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:00 smithi189 ceph-mon[30629]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:01.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:00 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:00 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:00 smithi189 ceph-mon[30629]: overall HEALTH_OK 2024-03-24T11:50:01.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:00 smithi189 ceph-mon[30629]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-24T11:50:01.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:00 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:00 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:00 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:01.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:00 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.208 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:00 smithi130 ceph-mon[25795]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:01.208 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:00 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.208 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:00 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.208 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:00 smithi130 ceph-mon[25795]: overall HEALTH_OK 2024-03-24T11:50:01.208 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:00 smithi130 ceph-mon[25795]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-24T11:50:01.208 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:00 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.208 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:00 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.208 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:00 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:01.208 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:00 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:00 smithi174 ceph-mon[30177]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:01.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:00 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:00 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:00 smithi174 ceph-mon[30177]: overall HEALTH_OK 2024-03-24T11:50:01.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:00 smithi174 ceph-mon[30177]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-24T11:50:01.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:00 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:00 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:00 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:01.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:00 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:01.819 INFO:teuthology.orchestra.run.smithi130.stdout:34359738389 2024-03-24T11:50:02.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:01 smithi189 ceph-mon[30629]: Adjusting osd_memory_target on smithi174 to 16064M 2024-03-24T11:50:02.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:01 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/1250103823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:50:02.204 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:01 smithi130 ceph-mon[25795]: Adjusting osd_memory_target on smithi174 to 16064M 2024-03-24T11:50:02.204 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:01 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/1250103823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:50:02.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:01 smithi174 ceph-mon[30177]: Adjusting osd_memory_target on smithi174 to 16064M 2024-03-24T11:50:02.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:01 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/1250103823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:50:02.500 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738388 got 34359738389 for osd.0 2024-03-24T11:50:02.500 DEBUG:teuthology.parallel:result is None 2024-03-24T11:50:02.847 INFO:teuthology.orchestra.run.smithi130.stdout:55834574865 2024-03-24T11:50:03.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:02 smithi130 ceph-mon[25795]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:03.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:02 smithi130 ceph-mon[25795]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-24T11:50:03.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:02 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/2451793386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:50:03.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:02 smithi189 ceph-mon[30629]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:03.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:02 smithi189 ceph-mon[30629]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-24T11:50:03.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:02 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/2451793386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:50:03.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:02 smithi174 ceph-mon[30177]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:03.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:02 smithi174 ceph-mon[30177]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-24T11:50:03.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:02 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/2451793386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:50:03.939 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574864 got 55834574865 for osd.1 2024-03-24T11:50:03.939 DEBUG:teuthology.parallel:result is None 2024-03-24T11:50:05.114 INFO:teuthology.orchestra.run.smithi130.stdout:77309411339 2024-03-24T11:50:05.287 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:04 smithi130 ceph-mon[25795]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:05.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:04 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:05.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:04 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:05.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:04 smithi174 ceph-mon[30177]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:05.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:04 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:05.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:04 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:05.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:04 smithi189 ceph-mon[30629]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:05.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:04 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:05.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:04 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:05.723 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411338 got 77309411339 for osd.2 2024-03-24T11:50:05.723 DEBUG:teuthology.parallel:result is None 2024-03-24T11:50:05.724 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T11:50:05.724 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:50:06.018 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:06.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:05 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/1576079966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:50:06.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:05 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:06.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:05 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:06.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:05 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:06.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:05 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:06.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:05 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:06.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:05 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:06.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:05 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/1576079966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:50:06.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:05 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:06.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:05 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:06.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:05 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:06.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:05 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:06.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:05 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:06.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:05 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:06.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:05 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/1576079966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:50:06.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:05 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:06.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:05 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:06.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:05 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:50:06.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:05 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:06.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:05 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:50:06.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:05 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:50:07.250 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:06 smithi130 ceph-mon[25795]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T11:50:07.250 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:06 smithi130 ceph-mon[25795]: Adjusting osd_memory_target on smithi130 to 16064M 2024-03-24T11:50:07.250 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:06 smithi130 ceph-mon[25795]: Updating smithi130:/etc/ceph/ceph.conf 2024-03-24T11:50:07.250 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:06 smithi130 ceph-mon[25795]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-24T11:50:07.250 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:06 smithi130 ceph-mon[25795]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:50:07.250 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:06 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:07.250 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:06 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:07.250 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:06 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:07.250 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:06 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:07.251 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:06 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: ::ffff:172.21.15.174 - - [24/Mar/2024:11:50:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:50:07.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:06 smithi174 ceph-mon[30177]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T11:50:07.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:06 smithi174 ceph-mon[30177]: Adjusting osd_memory_target on smithi130 to 16064M 2024-03-24T11:50:07.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:06 smithi174 ceph-mon[30177]: Updating smithi130:/etc/ceph/ceph.conf 2024-03-24T11:50:07.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:06 smithi174 ceph-mon[30177]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-24T11:50:07.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:06 smithi174 ceph-mon[30177]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:50:07.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:06 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:07.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:06 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:07.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:06 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:07.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:06 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:07.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:06 smithi189 ceph-mon[30629]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T11:50:07.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:06 smithi189 ceph-mon[30629]: Adjusting osd_memory_target on smithi130 to 16064M 2024-03-24T11:50:07.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:06 smithi189 ceph-mon[30629]: Updating smithi130:/etc/ceph/ceph.conf 2024-03-24T11:50:07.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:06 smithi189 ceph-mon[30629]: Updating smithi174:/etc/ceph/ceph.conf 2024-03-24T11:50:07.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:06 smithi189 ceph-mon[30629]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:50:07.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:06 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:07.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:06 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:07.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:06 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:07.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:06 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:08.001 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:50:08.003 INFO:teuthology.orchestra.run.smithi130.stderr:dumped all 2024-03-24T11:50:08.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: Updating smithi130:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:50:08.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:50:08.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:50:08.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:08.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:08.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:08.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: Updating smithi130:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:50:08.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:50:08.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:50:08.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:08.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:08.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:08.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:07 smithi189 ceph-mon[30629]: Deploying daemon alertmanager.a on smithi130 2024-03-24T11:50:08.287 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: Updating smithi130:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:50:08.287 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:50:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:50:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: Updating smithi130:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:50:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:50:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:50:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:08.288 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:07 smithi130 ceph-mon[25795]: Deploying daemon alertmanager.a on smithi130 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: Updating smithi130:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: Updating smithi130:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: Updating smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: Updating smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:08.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:07 smithi174 ceph-mon[30177]: Deploying daemon alertmanager.a on smithi130 2024-03-24T11:50:08.775 INFO:teuthology.orchestra.run.smithi130.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-03-24T11:50:07.150769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96648,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126520,"statfs":{"total":287972524032,"available":287873556480,"internally_reserved":0,"allocated":3182592,"data_stored":2109504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.996622"},"pg_stats":[{"pgid":"5.18","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138519+0000","last_change":"2024-03-24T11:49:32.882079+0000","last_active":"2024-03-24T11:49:57.138519+0000","last_peered":"2024-03-24T11:49:57.138519+0000","last_clean":"2024-03-24T11:49:57.138519+0000","last_became_active":"2024-03-24T11:49:32.881514+0000","last_became_peered":"2024-03-24T11:49:32.881514+0000","last_unstale":"2024-03-24T11:49:57.138519+0000","last_undegraded":"2024-03-24T11:49:57.138519+0000","last_fullsized":"2024-03-24T11:49:57.138519+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:58:33.328337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138556+0000","last_change":"2024-03-24T11:49:30.865209+0000","last_active":"2024-03-24T11:49:57.138556+0000","last_peered":"2024-03-24T11:49:57.138556+0000","last_clean":"2024-03-24T11:49:57.138556+0000","last_became_active":"2024-03-24T11:49:30.864715+0000","last_became_peered":"2024-03-24T11:49:30.864715+0000","last_unstale":"2024-03-24T11:49:57.138556+0000","last_undegraded":"2024-03-24T11:49:57.138556+0000","last_fullsized":"2024-03-24T11:49:57.138556+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:21:35.244233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.142355+0000","last_change":"2024-03-24T11:49:26.837779+0000","last_active":"2024-03-24T11:49:57.142355+0000","last_peered":"2024-03-24T11:49:57.142355+0000","last_clean":"2024-03-24T11:49:57.142355+0000","last_became_active":"2024-03-24T11:49:26.837496+0000","last_became_peered":"2024-03-24T11:49:26.837496+0000","last_unstale":"2024-03-24T11:49:57.142355+0000","last_undegraded":"2024-03-24T11:49:57.142355+0000","last_fullsized":"2024-03-24T11:49:57.142355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:03:39.447745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.142014+0000","last_change":"2024-03-24T11:49:28.852718+0000","last_active":"2024-03-24T11:49:57.142014+0000","last_peered":"2024-03-24T11:49:57.142014+0000","last_clean":"2024-03-24T11:49:57.142014+0000","last_became_active":"2024-03-24T11:49:28.852394+0000","last_became_peered":"2024-03-24T11:49:28.852394+0000","last_unstale":"2024-03-24T11:49:57.142014+0000","last_undegraded":"2024-03-24T11:49:57.142014+0000","last_fullsized":"2024-03-24T11:49:57.142014+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:59:11.078342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139744+0000","last_change":"2024-03-24T11:49:32.878867+0000","last_active":"2024-03-24T11:49:57.139744+0000","last_peered":"2024-03-24T11:49:57.139744+0000","last_clean":"2024-03-24T11:49:57.139744+0000","last_became_active":"2024-03-24T11:49:32.878508+0000","last_became_peered":"2024-03-24T11:49:32.878508+0000","last_unstale":"2024-03-24T11:49:57.139744+0000","last_undegraded":"2024-03-24T11:49:57.139744+0000","last_fullsized":"2024-03-24T11:49:57.139744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:46:43.266909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138975+0000","last_change":"2024-03-24T11:49:30.866542+0000","last_active":"2024-03-24T11:49:57.138975+0000","last_peered":"2024-03-24T11:49:57.138975+0000","last_clean":"2024-03-24T11:49:57.138975+0000","last_became_active":"2024-03-24T11:49:30.865898+0000","last_became_peered":"2024-03-24T11:49:30.865898+0000","last_unstale":"2024-03-24T11:49:57.138975+0000","last_undegraded":"2024-03-24T11:49:57.138975+0000","last_fullsized":"2024-03-24T11:49:57.138975+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:57.781307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141859+0000","last_change":"2024-03-24T11:49:26.837746+0000","last_active":"2024-03-24T11:49:57.141859+0000","last_peered":"2024-03-24T11:49:57.141859+0000","last_clean":"2024-03-24T11:49:57.141859+0000","last_became_active":"2024-03-24T11:49:26.837266+0000","last_became_peered":"2024-03-24T11:49:26.837266+0000","last_unstale":"2024-03-24T11:49:57.141859+0000","last_undegraded":"2024-03-24T11:49:57.141859+0000","last_fullsized":"2024-03-24T11:49:57.141859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:06.966471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.142221+0000","last_change":"2024-03-24T11:49:28.851532+0000","last_active":"2024-03-24T11:49:57.142221+0000","last_peered":"2024-03-24T11:49:57.142221+0000","last_clean":"2024-03-24T11:49:57.142221+0000","last_became_active":"2024-03-24T11:49:28.850989+0000","last_became_peered":"2024-03-24T11:49:28.850989+0000","last_unstale":"2024-03-24T11:49:57.142221+0000","last_undegraded":"2024-03-24T11:49:57.142221+0000","last_fullsized":"2024-03-24T11:49:57.142221+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:05:19.369840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138707+0000","last_change":"2024-03-24T11:49:32.881307+0000","last_active":"2024-03-24T11:49:57.138707+0000","last_peered":"2024-03-24T11:49:57.138707+0000","last_clean":"2024-03-24T11:49:57.138707+0000","last_became_active":"2024-03-24T11:49:32.881022+0000","last_became_peered":"2024-03-24T11:49:32.881022+0000","last_unstale":"2024-03-24T11:49:57.138707+0000","last_undegraded":"2024-03-24T11:49:57.138707+0000","last_fullsized":"2024-03-24T11:49:57.138707+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:57:28.804952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139337+0000","last_change":"2024-03-24T11:49:30.860570+0000","last_active":"2024-03-24T11:49:57.139337+0000","last_peered":"2024-03-24T11:49:57.139337+0000","last_clean":"2024-03-24T11:49:57.139337+0000","last_became_active":"2024-03-24T11:49:30.859647+0000","last_became_peered":"2024-03-24T11:49:30.859647+0000","last_unstale":"2024-03-24T11:49:57.139337+0000","last_undegraded":"2024-03-24T11:49:57.139337+0000","last_fullsized":"2024-03-24T11:49:57.139337+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:25.166269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141588+0000","last_change":"2024-03-24T11:49:26.835507+0000","last_active":"2024-03-24T11:49:57.141588+0000","last_peered":"2024-03-24T11:49:57.141588+0000","last_clean":"2024-03-24T11:49:57.141588+0000","last_became_active":"2024-03-24T11:49:26.834725+0000","last_became_peered":"2024-03-24T11:49:26.834725+0000","last_unstale":"2024-03-24T11:49:57.141588+0000","last_undegraded":"2024-03-24T11:49:57.141588+0000","last_fullsized":"2024-03-24T11:49:57.141588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:27.310999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140777+0000","last_change":"2024-03-24T11:49:28.850310+0000","last_active":"2024-03-24T11:49:57.140777+0000","last_peered":"2024-03-24T11:49:57.140777+0000","last_clean":"2024-03-24T11:49:57.140777+0000","last_became_active":"2024-03-24T11:49:28.848953+0000","last_became_peered":"2024-03-24T11:49:28.848953+0000","last_unstale":"2024-03-24T11:49:57.140777+0000","last_undegraded":"2024-03-24T11:49:57.140777+0000","last_fullsized":"2024-03-24T11:49:57.140777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:33:13.084663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139120+0000","last_change":"2024-03-24T11:49:32.881821+0000","last_active":"2024-03-24T11:49:57.139120+0000","last_peered":"2024-03-24T11:49:57.139120+0000","last_clean":"2024-03-24T11:49:57.139120+0000","last_became_active":"2024-03-24T11:49:32.881276+0000","last_became_peered":"2024-03-24T11:49:32.881276+0000","last_unstale":"2024-03-24T11:49:57.139120+0000","last_undegraded":"2024-03-24T11:49:57.139120+0000","last_fullsized":"2024-03-24T11:49:57.139120+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:19:06.054454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138938+0000","last_change":"2024-03-24T11:49:30.864629+0000","last_active":"2024-03-24T11:49:57.138938+0000","last_peered":"2024-03-24T11:49:57.138938+0000","last_clean":"2024-03-24T11:49:57.138938+0000","last_became_active":"2024-03-24T11:49:30.864062+0000","last_became_peered":"2024-03-24T11:49:30.864062+0000","last_unstale":"2024-03-24T11:49:57.138938+0000","last_undegraded":"2024-03-24T11:49:57.138938+0000","last_fullsized":"2024-03-24T11:49:57.138938+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:22:47.412210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140304+0000","last_change":"2024-03-24T11:49:26.837098+0000","last_active":"2024-03-24T11:49:57.140304+0000","last_peered":"2024-03-24T11:49:57.140304+0000","last_clean":"2024-03-24T11:49:57.140304+0000","last_became_active":"2024-03-24T11:49:26.836737+0000","last_became_peered":"2024-03-24T11:49:26.836737+0000","last_unstale":"2024-03-24T11:49:57.140304+0000","last_undegraded":"2024-03-24T11:49:57.140304+0000","last_fullsized":"2024-03-24T11:49:57.140304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:53:38.719175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141486+0000","last_change":"2024-03-24T11:49:28.850595+0000","last_active":"2024-03-24T11:49:57.141486+0000","last_peered":"2024-03-24T11:49:57.141486+0000","last_clean":"2024-03-24T11:49:57.141486+0000","last_became_active":"2024-03-24T11:49:28.850122+0000","last_became_peered":"2024-03-24T11:49:28.850122+0000","last_unstale":"2024-03-24T11:49:57.141486+0000","last_undegraded":"2024-03-24T11:49:57.141486+0000","last_fullsized":"2024-03-24T11:49:57.141486+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:41:23.234445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140244+0000","last_change":"2024-03-24T11:49:32.883293+0000","last_active":"2024-03-24T11:49:57.140244+0000","last_peered":"2024-03-24T11:49:57.140244+0000","last_clean":"2024-03-24T11:49:57.140244+0000","last_became_active":"2024-03-24T11:49:32.882892+0000","last_became_peered":"2024-03-24T11:49:32.882892+0000","last_unstale":"2024-03-24T11:49:57.140244+0000","last_undegraded":"2024-03-24T11:49:57.140244+0000","last_fullsized":"2024-03-24T11:49:57.140244+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:59.428492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140064+0000","last_change":"2024-03-24T11:49:30.864544+0000","last_active":"2024-03-24T11:49:57.140064+0000","last_peered":"2024-03-24T11:49:57.140064+0000","last_clean":"2024-03-24T11:49:57.140064+0000","last_became_active":"2024-03-24T11:49:30.864240+0000","last_became_peered":"2024-03-24T11:49:30.864240+0000","last_unstale":"2024-03-24T11:49:57.140064+0000","last_undegraded":"2024-03-24T11:49:57.140064+0000","last_fullsized":"2024-03-24T11:49:57.140064+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:12:02.582475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140004+0000","last_change":"2024-03-24T11:49:26.834671+0000","last_active":"2024-03-24T11:49:57.140004+0000","last_peered":"2024-03-24T11:49:57.140004+0000","last_clean":"2024-03-24T11:49:57.140004+0000","last_became_active":"2024-03-24T11:49:26.834255+0000","last_became_peered":"2024-03-24T11:49:26.834255+0000","last_unstale":"2024-03-24T11:49:57.140004+0000","last_undegraded":"2024-03-24T11:49:57.140004+0000","last_fullsized":"2024-03-24T11:49:57.140004+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:19:36.215970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139543+0000","last_change":"2024-03-24T11:49:28.851080+0000","last_active":"2024-03-24T11:49:57.139543+0000","last_peered":"2024-03-24T11:49:57.139543+0000","last_clean":"2024-03-24T11:49:57.139543+0000","last_became_active":"2024-03-24T11:49:28.850705+0000","last_became_peered":"2024-03-24T11:49:28.850705+0000","last_unstale":"2024-03-24T11:49:57.139543+0000","last_undegraded":"2024-03-24T11:49:57.139543+0000","last_fullsized":"2024-03-24T11:49:57.139543+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:48.230899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141608+0000","last_change":"2024-03-24T11:49:32.885182+0000","last_active":"2024-03-24T11:49:57.141608+0000","last_peered":"2024-03-24T11:49:57.141608+0000","last_clean":"2024-03-24T11:49:57.141608+0000","last_became_active":"2024-03-24T11:49:32.884860+0000","last_became_peered":"2024-03-24T11:49:32.884860+0000","last_unstale":"2024-03-24T11:49:57.141608+0000","last_undegraded":"2024-03-24T11:49:57.141608+0000","last_fullsized":"2024-03-24T11:49:57.141608+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:01:48.866841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140185+0000","last_change":"2024-03-24T11:49:30.862551+0000","last_active":"2024-03-24T11:49:57.140185+0000","last_peered":"2024-03-24T11:49:57.140185+0000","last_clean":"2024-03-24T11:49:57.140185+0000","last_became_active":"2024-03-24T11:49:30.861646+0000","last_became_peered":"2024-03-24T11:49:30.861646+0000","last_unstale":"2024-03-24T11:49:57.140185+0000","last_undegraded":"2024-03-24T11:49:57.140185+0000","last_fullsized":"2024-03-24T11:49:57.140185+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:08:55.361053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139615+0000","last_change":"2024-03-24T11:49:26.833093+0000","last_active":"2024-03-24T11:49:57.139615+0000","last_peered":"2024-03-24T11:49:57.139615+0000","last_clean":"2024-03-24T11:49:57.139615+0000","last_became_active":"2024-03-24T11:49:26.832600+0000","last_became_peered":"2024-03-24T11:49:26.832600+0000","last_unstale":"2024-03-24T11:49:57.139615+0000","last_undegraded":"2024-03-24T11:49:57.139615+0000","last_fullsized":"2024-03-24T11:49:57.139615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:02.898944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139627+0000","last_change":"2024-03-24T11:49:28.850890+0000","last_active":"2024-03-24T11:49:57.139627+0000","last_peered":"2024-03-24T11:49:57.139627+0000","last_clean":"2024-03-24T11:49:57.139627+0000","last_became_active":"2024-03-24T11:49:28.849717+0000","last_became_peered":"2024-03-24T11:49:28.849717+0000","last_unstale":"2024-03-24T11:49:57.139627+0000","last_undegraded":"2024-03-24T11:49:57.139627+0000","last_fullsized":"2024-03-24T11:49:57.139627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:09:15.966208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140631+0000","last_change":"2024-03-24T11:49:32.884470+0000","last_active":"2024-03-24T11:49:57.140631+0000","last_peered":"2024-03-24T11:49:57.140631+0000","last_clean":"2024-03-24T11:49:57.140631+0000","last_became_active":"2024-03-24T11:49:32.884036+0000","last_became_peered":"2024-03-24T11:49:32.884036+0000","last_unstale":"2024-03-24T11:49:57.140631+0000","last_undegraded":"2024-03-24T11:49:57.140631+0000","last_fullsized":"2024-03-24T11:49:57.140631+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:43:08.044397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140364+0000","last_change":"2024-03-24T11:49:30.859156+0000","last_active":"2024-03-24T11:49:57.140364+0000","last_peered":"2024-03-24T11:49:57.140364+0000","last_clean":"2024-03-24T11:49:57.140364+0000","last_became_active":"2024-03-24T11:49:30.858799+0000","last_became_peered":"2024-03-24T11:49:30.858799+0000","last_unstale":"2024-03-24T11:49:57.140364+0000","last_undegraded":"2024-03-24T11:49:57.140364+0000","last_fullsized":"2024-03-24T11:49:57.140364+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:17:30.150902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140292+0000","last_change":"2024-03-24T11:49:26.837563+0000","last_active":"2024-03-24T11:49:57.140292+0000","last_peered":"2024-03-24T11:49:57.140292+0000","last_clean":"2024-03-24T11:49:57.140292+0000","last_became_active":"2024-03-24T11:49:26.837241+0000","last_became_peered":"2024-03-24T11:49:26.837241+0000","last_unstale":"2024-03-24T11:49:57.140292+0000","last_undegraded":"2024-03-24T11:49:57.140292+0000","last_fullsized":"2024-03-24T11:49:57.140292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:51:21.736037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139285+0000","last_change":"2024-03-24T11:49:28.854103+0000","last_active":"2024-03-24T11:49:57.139285+0000","last_peered":"2024-03-24T11:49:57.139285+0000","last_clean":"2024-03-24T11:49:57.139285+0000","last_became_active":"2024-03-24T11:49:28.853327+0000","last_became_peered":"2024-03-24T11:49:28.853327+0000","last_unstale":"2024-03-24T11:49:57.139285+0000","last_undegraded":"2024-03-24T11:49:57.139285+0000","last_fullsized":"2024-03-24T11:49:57.139285+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:57:33.372728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140310+0000","last_change":"2024-03-24T11:49:32.882956+0000","last_active":"2024-03-24T11:49:57.140310+0000","last_peered":"2024-03-24T11:49:57.140310+0000","last_clean":"2024-03-24T11:49:57.140310+0000","last_became_active":"2024-03-24T11:49:32.882540+0000","last_became_peered":"2024-03-24T11:49:32.882540+0000","last_unstale":"2024-03-24T11:49:57.140310+0000","last_undegraded":"2024-03-24T11:49:57.140310+0000","last_fullsized":"2024-03-24T11:49:57.140310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:46:27.366070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140341+0000","last_change":"2024-03-24T11:49:30.868423+0000","last_active":"2024-03-24T11:49:57.140341+0000","last_peered":"2024-03-24T11:49:57.140341+0000","last_clean":"2024-03-24T11:49:57.140341+0000","last_became_active":"2024-03-24T11:49:30.867736+0000","last_became_peered":"2024-03-24T11:49:30.867736+0000","last_unstale":"2024-03-24T11:49:57.140341+0000","last_undegraded":"2024-03-24T11:49:57.140341+0000","last_fullsized":"2024-03-24T11:49:57.140341+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:00:56.171282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138797+0000","last_change":"2024-03-24T11:49:26.837803+0000","last_active":"2024-03-24T11:49:57.138797+0000","last_peered":"2024-03-24T11:49:57.138797+0000","last_clean":"2024-03-24T11:49:57.138797+0000","last_became_active":"2024-03-24T11:49:26.837230+0000","last_became_peered":"2024-03-24T11:49:26.837230+0000","last_unstale":"2024-03-24T11:49:57.138797+0000","last_undegraded":"2024-03-24T11:49:57.138797+0000","last_fullsized":"2024-03-24T11:49:57.138797+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:20:38.306158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139593+0000","last_change":"2024-03-24T11:49:28.850022+0000","last_active":"2024-03-24T11:49:57.139593+0000","last_peered":"2024-03-24T11:49:57.139593+0000","last_clean":"2024-03-24T11:49:57.139593+0000","last_became_active":"2024-03-24T11:49:28.849624+0000","last_became_peered":"2024-03-24T11:49:28.849624+0000","last_unstale":"2024-03-24T11:49:57.139593+0000","last_undegraded":"2024-03-24T11:49:57.139593+0000","last_fullsized":"2024-03-24T11:49:57.139593+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:19.627510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139233+0000","last_change":"2024-03-24T11:49:32.883178+0000","last_active":"2024-03-24T11:49:57.139233+0000","last_peered":"2024-03-24T11:49:57.139233+0000","last_clean":"2024-03-24T11:49:57.139233+0000","last_became_active":"2024-03-24T11:49:32.882455+0000","last_became_peered":"2024-03-24T11:49:32.882455+0000","last_unstale":"2024-03-24T11:49:57.139233+0000","last_undegraded":"2024-03-24T11:49:57.139233+0000","last_fullsized":"2024-03-24T11:49:57.139233+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:49:02.276165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139236+0000","last_change":"2024-03-24T11:49:30.869002+0000","last_active":"2024-03-24T11:49:57.139236+0000","last_peered":"2024-03-24T11:49:57.139236+0000","last_clean":"2024-03-24T11:49:57.139236+0000","last_became_active":"2024-03-24T11:49:30.868697+0000","last_became_peered":"2024-03-24T11:49:30.868697+0000","last_unstale":"2024-03-24T11:49:57.139236+0000","last_undegraded":"2024-03-24T11:49:57.139236+0000","last_fullsized":"2024-03-24T11:49:57.139236+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:19.305750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141461+0000","last_change":"2024-03-24T11:49:26.831136+0000","last_active":"2024-03-24T11:49:57.141461+0000","last_peered":"2024-03-24T11:49:57.141461+0000","last_clean":"2024-03-24T11:49:57.141461+0000","last_became_active":"2024-03-24T11:49:26.830219+0000","last_became_peered":"2024-03-24T11:49:26.830219+0000","last_unstale":"2024-03-24T11:49:57.141461+0000","last_undegraded":"2024-03-24T11:49:57.141461+0000","last_fullsized":"2024-03-24T11:49:57.141461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:38:31.342476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141113+0000","last_change":"2024-03-24T11:49:28.850101+0000","last_active":"2024-03-24T11:49:57.141113+0000","last_peered":"2024-03-24T11:49:57.141113+0000","last_clean":"2024-03-24T11:49:57.141113+0000","last_became_active":"2024-03-24T11:49:28.849021+0000","last_became_peered":"2024-03-24T11:49:28.849021+0000","last_unstale":"2024-03-24T11:49:57.141113+0000","last_undegraded":"2024-03-24T11:49:57.141113+0000","last_fullsized":"2024-03-24T11:49:57.141113+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:42:21.864632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138952+0000","last_change":"2024-03-24T11:49:32.880694+0000","last_active":"2024-03-24T11:49:57.138952+0000","last_peered":"2024-03-24T11:49:57.138952+0000","last_clean":"2024-03-24T11:49:57.138952+0000","last_became_active":"2024-03-24T11:49:32.880288+0000","last_became_peered":"2024-03-24T11:49:32.880288+0000","last_unstale":"2024-03-24T11:49:57.138952+0000","last_undegraded":"2024-03-24T11:49:57.138952+0000","last_fullsized":"2024-03-24T11:49:57.138952+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:12:03.173738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139574+0000","last_change":"2024-03-24T11:49:30.867703+0000","last_active":"2024-03-24T11:49:57.139574+0000","last_peered":"2024-03-24T11:49:57.139574+0000","last_clean":"2024-03-24T11:49:57.139574+0000","last_became_active":"2024-03-24T11:49:30.866776+0000","last_became_peered":"2024-03-24T11:49:30.866776+0000","last_unstale":"2024-03-24T11:49:57.139574+0000","last_undegraded":"2024-03-24T11:49:57.139574+0000","last_fullsized":"2024-03-24T11:49:57.139574+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:11:18.232452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140853+0000","last_change":"2024-03-24T11:49:26.831215+0000","last_active":"2024-03-24T11:49:57.140853+0000","last_peered":"2024-03-24T11:49:57.140853+0000","last_clean":"2024-03-24T11:49:57.140853+0000","last_became_active":"2024-03-24T11:49:26.830189+0000","last_became_peered":"2024-03-24T11:49:26.830189+0000","last_unstale":"2024-03-24T11:49:57.140853+0000","last_undegraded":"2024-03-24T11:49:57.140853+0000","last_fullsized":"2024-03-24T11:49:57.140853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:54:01.277105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.143323+0000","last_change":"2024-03-24T11:49:28.853115+0000","last_active":"2024-03-24T11:49:57.143323+0000","last_peered":"2024-03-24T11:49:57.143323+0000","last_clean":"2024-03-24T11:49:57.143323+0000","last_became_active":"2024-03-24T11:49:28.852529+0000","last_became_peered":"2024-03-24T11:49:28.852529+0000","last_unstale":"2024-03-24T11:49:57.143323+0000","last_undegraded":"2024-03-24T11:49:57.143323+0000","last_fullsized":"2024-03-24T11:49:57.143323+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:31:40.725251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138877+0000","last_change":"2024-03-24T11:49:32.880745+0000","last_active":"2024-03-24T11:49:57.138877+0000","last_peered":"2024-03-24T11:49:57.138877+0000","last_clean":"2024-03-24T11:49:57.138877+0000","last_became_active":"2024-03-24T11:49:32.880456+0000","last_became_peered":"2024-03-24T11:49:32.880456+0000","last_unstale":"2024-03-24T11:49:57.138877+0000","last_undegraded":"2024-03-24T11:49:57.138877+0000","last_fullsized":"2024-03-24T11:49:57.138877+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:10.240334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139487+0000","last_change":"2024-03-24T11:49:30.860422+0000","last_active":"2024-03-24T11:49:57.139487+0000","last_peered":"2024-03-24T11:49:57.139487+0000","last_clean":"2024-03-24T11:49:57.139487+0000","last_became_active":"2024-03-24T11:49:30.859575+0000","last_became_peered":"2024-03-24T11:49:30.859575+0000","last_unstale":"2024-03-24T11:49:57.139487+0000","last_undegraded":"2024-03-24T11:49:57.139487+0000","last_fullsized":"2024-03-24T11:49:57.139487+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:49:22.363912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141455+0000","last_change":"2024-03-24T11:49:26.831378+0000","last_active":"2024-03-24T11:49:57.141455+0000","last_peered":"2024-03-24T11:49:57.141455+0000","last_clean":"2024-03-24T11:49:57.141455+0000","last_became_active":"2024-03-24T11:49:26.830841+0000","last_became_peered":"2024-03-24T11:49:26.830841+0000","last_unstale":"2024-03-24T11:49:57.141455+0000","last_undegraded":"2024-03-24T11:49:57.141455+0000","last_fullsized":"2024-03-24T11:49:57.141455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:28:22.944311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141160+0000","last_change":"2024-03-24T11:49:28.848238+0000","last_active":"2024-03-24T11:49:57.141160+0000","last_peered":"2024-03-24T11:49:57.141160+0000","last_clean":"2024-03-24T11:49:57.141160+0000","last_became_active":"2024-03-24T11:49:28.847880+0000","last_became_peered":"2024-03-24T11:49:28.847880+0000","last_unstale":"2024-03-24T11:49:57.141160+0000","last_undegraded":"2024-03-24T11:49:57.141160+0000","last_fullsized":"2024-03-24T11:49:57.141160+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:32:35.434942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138947+0000","last_change":"2024-03-24T11:49:32.881543+0000","last_active":"2024-03-24T11:49:57.138947+0000","last_peered":"2024-03-24T11:49:57.138947+0000","last_clean":"2024-03-24T11:49:57.138947+0000","last_became_active":"2024-03-24T11:49:32.881217+0000","last_became_peered":"2024-03-24T11:49:32.881217+0000","last_unstale":"2024-03-24T11:49:57.138947+0000","last_undegraded":"2024-03-24T11:49:57.138947+0000","last_fullsized":"2024-03-24T11:49:57.138947+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:52:15.052552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139833+0000","last_change":"2024-03-24T11:49:30.867578+0000","last_active":"2024-03-24T11:49:57.139833+0000","last_peered":"2024-03-24T11:49:57.139833+0000","last_clean":"2024-03-24T11:49:57.139833+0000","last_became_active":"2024-03-24T11:49:30.867258+0000","last_became_peered":"2024-03-24T11:49:30.867258+0000","last_unstale":"2024-03-24T11:49:57.139833+0000","last_undegraded":"2024-03-24T11:49:57.139833+0000","last_fullsized":"2024-03-24T11:49:57.139833+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:53.755298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140632+0000","last_change":"2024-03-24T11:49:26.835772+0000","last_active":"2024-03-24T11:49:57.140632+0000","last_peered":"2024-03-24T11:49:57.140632+0000","last_clean":"2024-03-24T11:49:57.140632+0000","last_became_active":"2024-03-24T11:49:26.835348+0000","last_became_peered":"2024-03-24T11:49:26.835348+0000","last_unstale":"2024-03-24T11:49:57.140632+0000","last_undegraded":"2024-03-24T11:49:57.140632+0000","last_fullsized":"2024-03-24T11:49:57.140632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:04:20.417857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141267+0000","last_change":"2024-03-24T11:49:28.851086+0000","last_active":"2024-03-24T11:49:57.141267+0000","last_peered":"2024-03-24T11:49:57.141267+0000","last_clean":"2024-03-24T11:49:57.141267+0000","last_became_active":"2024-03-24T11:49:28.850333+0000","last_became_peered":"2024-03-24T11:49:28.850333+0000","last_unstale":"2024-03-24T11:49:57.141267+0000","last_undegraded":"2024-03-24T11:49:57.141267+0000","last_fullsized":"2024-03-24T11:49:57.141267+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:50:19.711047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139672+0000","last_change":"2024-03-24T11:49:32.885050+0000","last_active":"2024-03-24T11:49:57.139672+0000","last_peered":"2024-03-24T11:49:57.139672+0000","last_clean":"2024-03-24T11:49:57.139672+0000","last_became_active":"2024-03-24T11:49:32.884583+0000","last_became_peered":"2024-03-24T11:49:32.884583+0000","last_unstale":"2024-03-24T11:49:57.139672+0000","last_undegraded":"2024-03-24T11:49:57.139672+0000","last_fullsized":"2024-03-24T11:49:57.139672+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:43.946425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141204+0000","last_change":"2024-03-24T11:49:30.862644+0000","last_active":"2024-03-24T11:49:57.141204+0000","last_peered":"2024-03-24T11:49:57.141204+0000","last_clean":"2024-03-24T11:49:57.141204+0000","last_became_active":"2024-03-24T11:49:30.862394+0000","last_became_peered":"2024-03-24T11:49:30.862394+0000","last_unstale":"2024-03-24T11:49:57.141204+0000","last_undegraded":"2024-03-24T11:49:57.141204+0000","last_fullsized":"2024-03-24T11:49:57.141204+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:07:42.628384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140026+0000","last_change":"2024-03-24T11:49:26.833292+0000","last_active":"2024-03-24T11:49:57.140026+0000","last_peered":"2024-03-24T11:49:57.140026+0000","last_clean":"2024-03-24T11:49:57.140026+0000","last_became_active":"2024-03-24T11:49:26.832980+0000","last_became_peered":"2024-03-24T11:49:26.832980+0000","last_unstale":"2024-03-24T11:49:57.140026+0000","last_undegraded":"2024-03-24T11:49:57.140026+0000","last_fullsized":"2024-03-24T11:49:57.140026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:54:32.094868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140998+0000","last_change":"2024-03-24T11:49:28.850489+0000","last_active":"2024-03-24T11:49:57.140998+0000","last_peered":"2024-03-24T11:49:57.140998+0000","last_clean":"2024-03-24T11:49:57.140998+0000","last_became_active":"2024-03-24T11:49:28.850160+0000","last_became_peered":"2024-03-24T11:49:28.850160+0000","last_unstale":"2024-03-24T11:49:57.140998+0000","last_undegraded":"2024-03-24T11:49:57.140998+0000","last_fullsized":"2024-03-24T11:49:57.140998+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:39:13.549678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139886+0000","last_change":"2024-03-24T11:49:32.881872+0000","last_active":"2024-03-24T11:49:57.139886+0000","last_peered":"2024-03-24T11:49:57.139886+0000","last_clean":"2024-03-24T11:49:57.139886+0000","last_became_active":"2024-03-24T11:49:32.881375+0000","last_became_peered":"2024-03-24T11:49:32.881375+0000","last_unstale":"2024-03-24T11:49:57.139886+0000","last_undegraded":"2024-03-24T11:49:57.139886+0000","last_fullsized":"2024-03-24T11:49:57.139886+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:46:10.592569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140047+0000","last_change":"2024-03-24T11:49:30.864462+0000","last_active":"2024-03-24T11:49:57.140047+0000","last_peered":"2024-03-24T11:49:57.140047+0000","last_clean":"2024-03-24T11:49:57.140047+0000","last_became_active":"2024-03-24T11:49:30.864181+0000","last_became_peered":"2024-03-24T11:49:30.864181+0000","last_unstale":"2024-03-24T11:49:57.140047+0000","last_undegraded":"2024-03-24T11:49:57.140047+0000","last_fullsized":"2024-03-24T11:49:57.140047+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:19.375051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139157+0000","last_change":"2024-03-24T11:49:26.834596+0000","last_active":"2024-03-24T11:49:57.139157+0000","last_peered":"2024-03-24T11:49:57.139157+0000","last_clean":"2024-03-24T11:49:57.139157+0000","last_became_active":"2024-03-24T11:49:26.833504+0000","last_became_peered":"2024-03-24T11:49:26.833504+0000","last_unstale":"2024-03-24T11:49:57.139157+0000","last_undegraded":"2024-03-24T11:49:57.139157+0000","last_fullsized":"2024-03-24T11:49:57.139157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:32:19.425691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139963+0000","last_change":"2024-03-24T11:49:28.848565+0000","last_active":"2024-03-24T11:49:57.139963+0000","last_peered":"2024-03-24T11:49:57.139963+0000","last_clean":"2024-03-24T11:49:57.139963+0000","last_became_active":"2024-03-24T11:49:28.848025+0000","last_became_peered":"2024-03-24T11:49:28.848025+0000","last_unstale":"2024-03-24T11:49:57.139963+0000","last_undegraded":"2024-03-24T11:49:57.139963+0000","last_fullsized":"2024-03-24T11:49:57.139963+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:11:40.434654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140274+0000","last_change":"2024-03-24T11:49:32.881554+0000","last_active":"2024-03-24T11:49:57.140274+0000","last_peered":"2024-03-24T11:49:57.140274+0000","last_clean":"2024-03-24T11:49:57.140274+0000","last_became_active":"2024-03-24T11:49:32.880618+0000","last_became_peered":"2024-03-24T11:49:32.880618+0000","last_unstale":"2024-03-24T11:49:57.140274+0000","last_undegraded":"2024-03-24T11:49:57.140274+0000","last_fullsized":"2024-03-24T11:49:57.140274+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:16.260169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.142122+0000","last_change":"2024-03-24T11:49:30.869188+0000","last_active":"2024-03-24T11:49:57.142122+0000","last_peered":"2024-03-24T11:49:57.142122+0000","last_clean":"2024-03-24T11:49:57.142122+0000","last_became_active":"2024-03-24T11:49:30.868877+0000","last_became_peered":"2024-03-24T11:49:30.868877+0000","last_unstale":"2024-03-24T11:49:57.142122+0000","last_undegraded":"2024-03-24T11:49:57.142122+0000","last_fullsized":"2024-03-24T11:49:57.142122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:26:51.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138940+0000","last_change":"2024-03-24T11:49:26.831243+0000","last_active":"2024-03-24T11:49:57.138940+0000","last_peered":"2024-03-24T11:49:57.138940+0000","last_clean":"2024-03-24T11:49:57.138940+0000","last_became_active":"2024-03-24T11:49:26.830403+0000","last_became_peered":"2024-03-24T11:49:26.830403+0000","last_unstale":"2024-03-24T11:49:57.138940+0000","last_undegraded":"2024-03-24T11:49:57.138940+0000","last_fullsized":"2024-03-24T11:49:57.138940+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:04:29.428630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138857+0000","last_change":"2024-03-24T11:49:28.850593+0000","last_active":"2024-03-24T11:49:57.138857+0000","last_peered":"2024-03-24T11:49:57.138857+0000","last_clean":"2024-03-24T11:49:57.138857+0000","last_became_active":"2024-03-24T11:49:28.850298+0000","last_became_peered":"2024-03-24T11:49:28.850298+0000","last_unstale":"2024-03-24T11:49:57.138857+0000","last_undegraded":"2024-03-24T11:49:57.138857+0000","last_fullsized":"2024-03-24T11:49:57.138857+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:29.430524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141987+0000","last_change":"2024-03-24T11:49:32.884153+0000","last_active":"2024-03-24T11:49:57.141987+0000","last_peered":"2024-03-24T11:49:57.141987+0000","last_clean":"2024-03-24T11:49:57.141987+0000","last_became_active":"2024-03-24T11:49:32.883704+0000","last_became_peered":"2024-03-24T11:49:32.883704+0000","last_unstale":"2024-03-24T11:49:57.141987+0000","last_undegraded":"2024-03-24T11:49:57.141987+0000","last_fullsized":"2024-03-24T11:49:57.141987+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:15:17.917106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140433+0000","last_change":"2024-03-24T11:49:30.868090+0000","last_active":"2024-03-24T11:49:57.140433+0000","last_peered":"2024-03-24T11:49:57.140433+0000","last_clean":"2024-03-24T11:49:57.140433+0000","last_became_active":"2024-03-24T11:49:30.867329+0000","last_became_peered":"2024-03-24T11:49:30.867329+0000","last_unstale":"2024-03-24T11:49:57.140433+0000","last_undegraded":"2024-03-24T11:49:57.140433+0000","last_fullsized":"2024-03-24T11:49:57.140433+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:57:17.526006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139047+0000","last_change":"2024-03-24T11:49:26.837022+0000","last_active":"2024-03-24T11:49:57.139047+0000","last_peered":"2024-03-24T11:49:57.139047+0000","last_clean":"2024-03-24T11:49:57.139047+0000","last_became_active":"2024-03-24T11:49:26.834790+0000","last_became_peered":"2024-03-24T11:49:26.834790+0000","last_unstale":"2024-03-24T11:49:57.139047+0000","last_undegraded":"2024-03-24T11:49:57.139047+0000","last_fullsized":"2024-03-24T11:49:57.139047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:33.082967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139078+0000","last_change":"2024-03-24T11:49:28.850485+0000","last_active":"2024-03-24T11:49:57.139078+0000","last_peered":"2024-03-24T11:49:57.139078+0000","last_clean":"2024-03-24T11:49:57.139078+0000","last_became_active":"2024-03-24T11:49:28.849659+0000","last_became_peered":"2024-03-24T11:49:28.849659+0000","last_unstale":"2024-03-24T11:49:57.139078+0000","last_undegraded":"2024-03-24T11:49:57.139078+0000","last_fullsized":"2024-03-24T11:49:57.139078+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:12:47.288853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139629+0000","last_change":"2024-03-24T11:49:32.884915+0000","last_active":"2024-03-24T11:49:57.139629+0000","last_peered":"2024-03-24T11:49:57.139629+0000","last_clean":"2024-03-24T11:49:57.139629+0000","last_became_active":"2024-03-24T11:49:32.884604+0000","last_became_peered":"2024-03-24T11:49:32.884604+0000","last_unstale":"2024-03-24T11:49:57.139629+0000","last_undegraded":"2024-03-24T11:49:57.139629+0000","last_fullsized":"2024-03-24T11:49:57.139629+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:43.313298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140506+0000","last_change":"2024-03-24T11:49:26.834688+0000","last_active":"2024-03-24T11:49:57.140506+0000","last_peered":"2024-03-24T11:49:57.140506+0000","last_clean":"2024-03-24T11:49:57.140506+0000","last_became_active":"2024-03-24T11:49:26.833929+0000","last_became_peered":"2024-03-24T11:49:26.833929+0000","last_unstale":"2024-03-24T11:49:57.140506+0000","last_undegraded":"2024-03-24T11:49:57.140506+0000","last_fullsized":"2024-03-24T11:49:57.140506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:03:52.862106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140704+0000","last_change":"2024-03-24T11:49:28.850282+0000","last_active":"2024-03-24T11:49:57.140704+0000","last_peered":"2024-03-24T11:49:57.140704+0000","last_clean":"2024-03-24T11:49:57.140704+0000","last_became_active":"2024-03-24T11:49:28.849994+0000","last_became_peered":"2024-03-24T11:49:28.849994+0000","last_unstale":"2024-03-24T11:49:57.140704+0000","last_undegraded":"2024-03-24T11:49:57.140704+0000","last_fullsized":"2024-03-24T11:49:57.140704+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:07:20.366273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138674+0000","last_change":"2024-03-24T11:49:30.859104+0000","last_active":"2024-03-24T11:49:57.138674+0000","last_peered":"2024-03-24T11:49:57.138674+0000","last_clean":"2024-03-24T11:49:57.138674+0000","last_became_active":"2024-03-24T11:49:30.858779+0000","last_became_peered":"2024-03-24T11:49:30.858779+0000","last_unstale":"2024-03-24T11:49:57.138674+0000","last_undegraded":"2024-03-24T11:49:57.138674+0000","last_fullsized":"2024-03-24T11:49:57.138674+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:51:45.455282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139759+0000","last_change":"2024-03-24T11:49:32.881363+0000","last_active":"2024-03-24T11:49:57.139759+0000","last_peered":"2024-03-24T11:49:57.139759+0000","last_clean":"2024-03-24T11:49:57.139759+0000","last_became_active":"2024-03-24T11:49:32.881023+0000","last_became_peered":"2024-03-24T11:49:32.881023+0000","last_unstale":"2024-03-24T11:49:57.139759+0000","last_undegraded":"2024-03-24T11:49:57.139759+0000","last_fullsized":"2024-03-24T11:49:57.139759+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:54:43.612364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141361+0000","last_change":"2024-03-24T11:49:26.840208+0000","last_active":"2024-03-24T11:49:57.141361+0000","last_peered":"2024-03-24T11:49:57.141361+0000","last_clean":"2024-03-24T11:49:57.141361+0000","last_became_active":"2024-03-24T11:49:26.839960+0000","last_became_peered":"2024-03-24T11:49:26.839960+0000","last_unstale":"2024-03-24T11:49:57.141361+0000","last_undegraded":"2024-03-24T11:49:57.141361+0000","last_fullsized":"2024-03-24T11:49:57.141361+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:05:59.317091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":53,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141094+0000","last_change":"2024-03-24T11:49:28.848929+0000","last_active":"2024-03-24T11:49:57.141094+0000","last_peered":"2024-03-24T11:49:57.141094+0000","last_clean":"2024-03-24T11:49:57.141094+0000","last_became_active":"2024-03-24T11:49:28.848494+0000","last_became_peered":"2024-03-24T11:49:28.848494+0000","last_unstale":"2024-03-24T11:49:57.141094+0000","last_undegraded":"2024-03-24T11:49:57.141094+0000","last_fullsized":"2024-03-24T11:49:57.141094+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:35:20.301258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138969+0000","last_change":"2024-03-24T11:49:30.860547+0000","last_active":"2024-03-24T11:49:57.138969+0000","last_peered":"2024-03-24T11:49:57.138969+0000","last_clean":"2024-03-24T11:49:57.138969+0000","last_became_active":"2024-03-24T11:49:30.859905+0000","last_became_peered":"2024-03-24T11:49:30.859905+0000","last_unstale":"2024-03-24T11:49:57.138969+0000","last_undegraded":"2024-03-24T11:49:57.138969+0000","last_fullsized":"2024-03-24T11:49:57.138969+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:29:46.368494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140290+0000","last_change":"2024-03-24T11:49:32.883820+0000","last_active":"2024-03-24T11:49:57.140290+0000","last_peered":"2024-03-24T11:49:57.140290+0000","last_clean":"2024-03-24T11:49:57.140290+0000","last_became_active":"2024-03-24T11:49:32.883334+0000","last_became_peered":"2024-03-24T11:49:32.883334+0000","last_unstale":"2024-03-24T11:49:57.140290+0000","last_undegraded":"2024-03-24T11:49:57.140290+0000","last_fullsized":"2024-03-24T11:49:57.140290+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:15.107756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139738+0000","last_change":"2024-03-24T11:49:26.831325+0000","last_active":"2024-03-24T11:49:57.139738+0000","last_peered":"2024-03-24T11:49:57.139738+0000","last_clean":"2024-03-24T11:49:57.139738+0000","last_became_active":"2024-03-24T11:49:26.830449+0000","last_became_peered":"2024-03-24T11:49:26.830449+0000","last_unstale":"2024-03-24T11:49:57.139738+0000","last_undegraded":"2024-03-24T11:49:57.139738+0000","last_fullsized":"2024-03-24T11:49:57.139738+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:42.502768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140401+0000","last_change":"2024-03-24T11:49:28.850435+0000","last_active":"2024-03-24T11:49:57.140401+0000","last_peered":"2024-03-24T11:49:57.140401+0000","last_clean":"2024-03-24T11:49:57.140401+0000","last_became_active":"2024-03-24T11:49:28.850162+0000","last_became_peered":"2024-03-24T11:49:28.850162+0000","last_unstale":"2024-03-24T11:49:57.140401+0000","last_undegraded":"2024-03-24T11:49:57.140401+0000","last_fullsized":"2024-03-24T11:49:57.140401+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:37:53.163817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140544+0000","last_change":"2024-03-24T11:49:30.867938+0000","last_active":"2024-03-24T11:49:57.140544+0000","last_peered":"2024-03-24T11:49:57.140544+0000","last_clean":"2024-03-24T11:49:57.140544+0000","last_became_active":"2024-03-24T11:49:30.867128+0000","last_became_peered":"2024-03-24T11:49:30.867128+0000","last_unstale":"2024-03-24T11:49:57.140544+0000","last_undegraded":"2024-03-24T11:49:57.140544+0000","last_fullsized":"2024-03-24T11:49:57.140544+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:48:47.967469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141896+0000","last_change":"2024-03-24T11:49:32.884428+0000","last_active":"2024-03-24T11:49:57.141896+0000","last_peered":"2024-03-24T11:49:57.141896+0000","last_clean":"2024-03-24T11:49:57.141896+0000","last_became_active":"2024-03-24T11:49:32.884000+0000","last_became_peered":"2024-03-24T11:49:32.884000+0000","last_unstale":"2024-03-24T11:49:57.141896+0000","last_undegraded":"2024-03-24T11:49:57.141896+0000","last_fullsized":"2024-03-24T11:49:57.141896+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:58:05.113766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'18","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138522+0000","last_change":"2024-03-24T11:49:28.851547+0000","last_active":"2024-03-24T11:49:57.138522+0000","last_peered":"2024-03-24T11:49:57.138522+0000","last_clean":"2024-03-24T11:49:57.138522+0000","last_became_active":"2024-03-24T11:49:28.851174+0000","last_became_peered":"2024-03-24T11:49:28.851174+0000","last_unstale":"2024-03-24T11:49:57.138522+0000","last_undegraded":"2024-03-24T11:49:57.138522+0000","last_fullsized":"2024-03-24T11:49:57.138522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:39:11.038726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140053+0000","last_change":"2024-03-24T11:49:26.835454+0000","last_active":"2024-03-24T11:49:57.140053+0000","last_peered":"2024-03-24T11:49:57.140053+0000","last_clean":"2024-03-24T11:49:57.140053+0000","last_became_active":"2024-03-24T11:49:26.835117+0000","last_became_peered":"2024-03-24T11:49:26.835117+0000","last_unstale":"2024-03-24T11:49:57.140053+0000","last_undegraded":"2024-03-24T11:49:57.140053+0000","last_fullsized":"2024-03-24T11:49:57.140053+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:09:50.187146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.143286+0000","last_change":"2024-03-24T11:49:30.869424+0000","last_active":"2024-03-24T11:49:57.143286+0000","last_peered":"2024-03-24T11:49:57.143286+0000","last_clean":"2024-03-24T11:49:57.143286+0000","last_became_active":"2024-03-24T11:49:30.869059+0000","last_became_peered":"2024-03-24T11:49:30.869059+0000","last_unstale":"2024-03-24T11:49:57.143286+0000","last_undegraded":"2024-03-24T11:49:57.143286+0000","last_fullsized":"2024-03-24T11:49:57.143286+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:03.244382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139672+0000","last_change":"2024-03-24T11:49:32.881811+0000","last_active":"2024-03-24T11:49:57.139672+0000","last_peered":"2024-03-24T11:49:57.139672+0000","last_clean":"2024-03-24T11:49:57.139672+0000","last_became_active":"2024-03-24T11:49:32.881493+0000","last_became_peered":"2024-03-24T11:49:32.881493+0000","last_unstale":"2024-03-24T11:49:57.139672+0000","last_undegraded":"2024-03-24T11:49:57.139672+0000","last_fullsized":"2024-03-24T11:49:57.139672+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:07:26.879042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141170+0000","last_change":"2024-03-24T11:49:28.851363+0000","last_active":"2024-03-24T11:49:57.141170+0000","last_peered":"2024-03-24T11:49:57.141170+0000","last_clean":"2024-03-24T11:49:57.141170+0000","last_became_active":"2024-03-24T11:49:28.850236+0000","last_became_peered":"2024-03-24T11:49:28.850236+0000","last_unstale":"2024-03-24T11:49:57.141170+0000","last_undegraded":"2024-03-24T11:49:57.141170+0000","last_fullsized":"2024-03-24T11:49:57.141170+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:21:17.101167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139974+0000","last_change":"2024-03-24T11:49:26.831102+0000","last_active":"2024-03-24T11:49:57.139974+0000","last_peered":"2024-03-24T11:49:57.139974+0000","last_clean":"2024-03-24T11:49:57.139974+0000","last_became_active":"2024-03-24T11:49:26.830182+0000","last_became_peered":"2024-03-24T11:49:26.830182+0000","last_unstale":"2024-03-24T11:49:57.139974+0000","last_undegraded":"2024-03-24T11:49:57.139974+0000","last_fullsized":"2024-03-24T11:49:57.139974+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:12:41.734925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:59.236671+0000","last_change":"2024-03-24T11:49:16.896490+0000","last_active":"2024-03-24T11:49:59.236671+0000","last_peered":"2024-03-24T11:49:59.236671+0000","last_clean":"2024-03-24T11:49:59.236671+0000","last_became_active":"2024-03-24T11:49:16.895549+0000","last_became_peered":"2024-03-24T11:49:16.895549+0000","last_unstale":"2024-03-24T11:49:59.236671+0000","last_undegraded":"2024-03-24T11:49:59.236671+0000","last_fullsized":"2024-03-24T11:49:59.236671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:15.880507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:15.880507+0000","last_clean_scrub_stamp":"2024-03-24T11:49:15.880507+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:02:42.558300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140837+0000","last_change":"2024-03-24T11:49:30.863746+0000","last_active":"2024-03-24T11:49:57.140837+0000","last_peered":"2024-03-24T11:49:57.140837+0000","last_clean":"2024-03-24T11:49:57.140837+0000","last_became_active":"2024-03-24T11:49:30.863357+0000","last_became_peered":"2024-03-24T11:49:30.863357+0000","last_unstale":"2024-03-24T11:49:57.140837+0000","last_undegraded":"2024-03-24T11:49:57.140837+0000","last_fullsized":"2024-03-24T11:49:57.140837+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:05:37.110571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140407+0000","last_change":"2024-03-24T11:49:32.883264+0000","last_active":"2024-03-24T11:49:57.140407+0000","last_peered":"2024-03-24T11:49:57.140407+0000","last_clean":"2024-03-24T11:49:57.140407+0000","last_became_active":"2024-03-24T11:49:32.882778+0000","last_became_peered":"2024-03-24T11:49:32.882778+0000","last_unstale":"2024-03-24T11:49:57.140407+0000","last_undegraded":"2024-03-24T11:49:57.140407+0000","last_fullsized":"2024-03-24T11:49:57.140407+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:44:06.868622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'14","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140481+0000","last_change":"2024-03-24T11:49:28.850342+0000","last_active":"2024-03-24T11:49:57.140481+0000","last_peered":"2024-03-24T11:49:57.140481+0000","last_clean":"2024-03-24T11:49:57.140481+0000","last_became_active":"2024-03-24T11:49:28.849889+0000","last_became_peered":"2024-03-24T11:49:28.849889+0000","last_unstale":"2024-03-24T11:49:57.140481+0000","last_undegraded":"2024-03-24T11:49:57.140481+0000","last_fullsized":"2024-03-24T11:49:57.140481+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:18:08.270045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138809+0000","last_change":"2024-03-24T11:49:26.835111+0000","last_active":"2024-03-24T11:49:57.138809+0000","last_peered":"2024-03-24T11:49:57.138809+0000","last_clean":"2024-03-24T11:49:57.138809+0000","last_became_active":"2024-03-24T11:49:26.834742+0000","last_became_peered":"2024-03-24T11:49:26.834742+0000","last_unstale":"2024-03-24T11:49:57.138809+0000","last_undegraded":"2024-03-24T11:49:57.138809+0000","last_fullsized":"2024-03-24T11:49:57.138809+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:58.677268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140518+0000","last_change":"2024-03-24T11:49:30.860150+0000","last_active":"2024-03-24T11:49:57.140518+0000","last_peered":"2024-03-24T11:49:57.140518+0000","last_clean":"2024-03-24T11:49:57.140518+0000","last_became_active":"2024-03-24T11:49:30.859839+0000","last_became_peered":"2024-03-24T11:49:30.859839+0000","last_unstale":"2024-03-24T11:49:57.140518+0000","last_undegraded":"2024-03-24T11:49:57.140518+0000","last_fullsized":"2024-03-24T11:49:57.140518+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:31:13.190445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139521+0000","last_change":"2024-03-24T11:49:32.881406+0000","last_active":"2024-03-24T11:49:57.139521+0000","last_peered":"2024-03-24T11:49:57.139521+0000","last_clean":"2024-03-24T11:49:57.139521+0000","last_became_active":"2024-03-24T11:49:32.881016+0000","last_became_peered":"2024-03-24T11:49:32.881016+0000","last_unstale":"2024-03-24T11:49:57.139521+0000","last_undegraded":"2024-03-24T11:49:57.139521+0000","last_fullsized":"2024-03-24T11:49:57.139521+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:10:49.300161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141370+0000","last_change":"2024-03-24T11:49:26.833033+0000","last_active":"2024-03-24T11:49:57.141370+0000","last_peered":"2024-03-24T11:49:57.141370+0000","last_clean":"2024-03-24T11:49:57.141370+0000","last_became_active":"2024-03-24T11:49:26.832733+0000","last_became_peered":"2024-03-24T11:49:26.832733+0000","last_unstale":"2024-03-24T11:49:57.141370+0000","last_undegraded":"2024-03-24T11:49:57.141370+0000","last_fullsized":"2024-03-24T11:49:57.141370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:58:42.173192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'28","reported_seq":79,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141192+0000","last_change":"2024-03-24T11:49:28.851594+0000","last_active":"2024-03-24T11:49:57.141192+0000","last_peered":"2024-03-24T11:49:57.141192+0000","last_clean":"2024-03-24T11:49:57.141192+0000","last_became_active":"2024-03-24T11:49:28.851268+0000","last_became_peered":"2024-03-24T11:49:28.851268+0000","last_unstale":"2024-03-24T11:49:57.141192+0000","last_undegraded":"2024-03-24T11:49:57.141192+0000","last_fullsized":"2024-03-24T11:49:57.141192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:50:23.249657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139547+0000","last_change":"2024-03-24T11:49:30.860075+0000","last_active":"2024-03-24T11:49:57.139547+0000","last_peered":"2024-03-24T11:49:57.139547+0000","last_clean":"2024-03-24T11:49:57.139547+0000","last_became_active":"2024-03-24T11:49:30.859742+0000","last_became_peered":"2024-03-24T11:49:30.859742+0000","last_unstale":"2024-03-24T11:49:57.139547+0000","last_undegraded":"2024-03-24T11:49:57.139547+0000","last_fullsized":"2024-03-24T11:49:57.139547+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:30:42.314971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138552+0000","last_change":"2024-03-24T11:49:32.882459+0000","last_active":"2024-03-24T11:49:57.138552+0000","last_peered":"2024-03-24T11:49:57.138552+0000","last_clean":"2024-03-24T11:49:57.138552+0000","last_became_active":"2024-03-24T11:49:32.881768+0000","last_became_peered":"2024-03-24T11:49:32.881768+0000","last_unstale":"2024-03-24T11:49:57.138552+0000","last_undegraded":"2024-03-24T11:49:57.138552+0000","last_fullsized":"2024-03-24T11:49:57.138552+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:28:06.013133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141664+0000","last_change":"2024-03-24T11:49:26.836256+0000","last_active":"2024-03-24T11:49:57.141664+0000","last_peered":"2024-03-24T11:49:57.141664+0000","last_clean":"2024-03-24T11:49:57.141664+0000","last_became_active":"2024-03-24T11:49:26.835548+0000","last_became_peered":"2024-03-24T11:49:26.835548+0000","last_unstale":"2024-03-24T11:49:57.141664+0000","last_undegraded":"2024-03-24T11:49:57.141664+0000","last_fullsized":"2024-03-24T11:49:57.141664+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:07:22.041247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.143436+0000","last_change":"2024-03-24T11:49:28.853307+0000","last_active":"2024-03-24T11:49:57.143436+0000","last_peered":"2024-03-24T11:49:57.143436+0000","last_clean":"2024-03-24T11:49:57.143436+0000","last_became_active":"2024-03-24T11:49:28.852918+0000","last_became_peered":"2024-03-24T11:49:28.852918+0000","last_unstale":"2024-03-24T11:49:57.143436+0000","last_undegraded":"2024-03-24T11:49:57.143436+0000","last_fullsized":"2024-03-24T11:49:57.143436+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:52:19.170963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139704+0000","last_change":"2024-03-24T11:49:30.864224+0000","last_active":"2024-03-24T11:49:57.139704+0000","last_peered":"2024-03-24T11:49:57.139704+0000","last_clean":"2024-03-24T11:49:57.139704+0000","last_became_active":"2024-03-24T11:49:30.863940+0000","last_became_peered":"2024-03-24T11:49:30.863940+0000","last_unstale":"2024-03-24T11:49:57.139704+0000","last_undegraded":"2024-03-24T11:49:57.139704+0000","last_fullsized":"2024-03-24T11:49:57.139704+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:32:17.040409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139533+0000","last_change":"2024-03-24T11:49:32.880662+0000","last_active":"2024-03-24T11:49:57.139533+0000","last_peered":"2024-03-24T11:49:57.139533+0000","last_clean":"2024-03-24T11:49:57.139533+0000","last_became_active":"2024-03-24T11:49:32.880384+0000","last_became_peered":"2024-03-24T11:49:32.880384+0000","last_unstale":"2024-03-24T11:49:57.139533+0000","last_undegraded":"2024-03-24T11:49:57.139533+0000","last_fullsized":"2024-03-24T11:49:57.139533+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:42:13.999971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141081+0000","last_change":"2024-03-24T11:49:26.834924+0000","last_active":"2024-03-24T11:49:57.141081+0000","last_peered":"2024-03-24T11:49:57.141081+0000","last_clean":"2024-03-24T11:49:57.141081+0000","last_became_active":"2024-03-24T11:49:26.834589+0000","last_became_peered":"2024-03-24T11:49:26.834589+0000","last_unstale":"2024-03-24T11:49:57.141081+0000","last_undegraded":"2024-03-24T11:49:57.141081+0000","last_fullsized":"2024-03-24T11:49:57.141081+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:24.738910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141472+0000","last_change":"2024-03-24T11:49:28.850377+0000","last_active":"2024-03-24T11:49:57.141472+0000","last_peered":"2024-03-24T11:49:57.141472+0000","last_clean":"2024-03-24T11:49:57.141472+0000","last_became_active":"2024-03-24T11:49:28.849991+0000","last_became_peered":"2024-03-24T11:49:28.849991+0000","last_unstale":"2024-03-24T11:49:57.141472+0000","last_undegraded":"2024-03-24T11:49:57.141472+0000","last_fullsized":"2024-03-24T11:49:57.141472+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:38:46.705905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138654+0000","last_change":"2024-03-24T11:49:30.859125+0000","last_active":"2024-03-24T11:49:57.138654+0000","last_peered":"2024-03-24T11:49:57.138654+0000","last_clean":"2024-03-24T11:49:57.138654+0000","last_became_active":"2024-03-24T11:49:30.858867+0000","last_became_peered":"2024-03-24T11:49:30.858867+0000","last_unstale":"2024-03-24T11:49:57.138654+0000","last_undegraded":"2024-03-24T11:49:57.138654+0000","last_fullsized":"2024-03-24T11:49:57.138654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:23:29.970739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140585+0000","last_change":"2024-03-24T11:49:32.882073+0000","last_active":"2024-03-24T11:49:57.140585+0000","last_peered":"2024-03-24T11:49:57.140585+0000","last_clean":"2024-03-24T11:49:57.140585+0000","last_became_active":"2024-03-24T11:49:32.881265+0000","last_became_peered":"2024-03-24T11:49:32.881265+0000","last_unstale":"2024-03-24T11:49:57.140585+0000","last_undegraded":"2024-03-24T11:49:57.140585+0000","last_fullsized":"2024-03-24T11:49:57.140585+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:01:23.527011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140585+0000","last_change":"2024-03-24T11:49:30.866180+0000","last_active":"2024-03-24T11:49:57.140585+0000","last_peered":"2024-03-24T11:49:57.140585+0000","last_clean":"2024-03-24T11:49:57.140585+0000","last_became_active":"2024-03-24T11:49:30.865820+0000","last_became_peered":"2024-03-24T11:49:30.865820+0000","last_unstale":"2024-03-24T11:49:57.140585+0000","last_undegraded":"2024-03-24T11:49:57.140585+0000","last_fullsized":"2024-03-24T11:49:57.140585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:07.099753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138753+0000","last_change":"2024-03-24T11:49:26.835688+0000","last_active":"2024-03-24T11:49:57.138753+0000","last_peered":"2024-03-24T11:49:57.138753+0000","last_clean":"2024-03-24T11:49:57.138753+0000","last_became_active":"2024-03-24T11:49:26.835244+0000","last_became_peered":"2024-03-24T11:49:26.835244+0000","last_unstale":"2024-03-24T11:49:57.138753+0000","last_undegraded":"2024-03-24T11:49:57.138753+0000","last_fullsized":"2024-03-24T11:49:57.138753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:39:39.385060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139216+0000","last_change":"2024-03-24T11:49:28.850122+0000","last_active":"2024-03-24T11:49:57.139216+0000","last_peered":"2024-03-24T11:49:57.139216+0000","last_clean":"2024-03-24T11:49:57.139216+0000","last_became_active":"2024-03-24T11:49:28.849817+0000","last_became_peered":"2024-03-24T11:49:28.849817+0000","last_unstale":"2024-03-24T11:49:57.139216+0000","last_undegraded":"2024-03-24T11:49:57.139216+0000","last_fullsized":"2024-03-24T11:49:57.139216+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:39:40.308805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141333+0000","last_change":"2024-03-24T11:49:32.882010+0000","last_active":"2024-03-24T11:49:57.141333+0000","last_peered":"2024-03-24T11:49:57.141333+0000","last_clean":"2024-03-24T11:49:57.141333+0000","last_became_active":"2024-03-24T11:49:32.881696+0000","last_became_peered":"2024-03-24T11:49:32.881696+0000","last_unstale":"2024-03-24T11:49:57.141333+0000","last_undegraded":"2024-03-24T11:49:57.141333+0000","last_fullsized":"2024-03-24T11:49:57.141333+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:58:35.897234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.142147+0000","last_change":"2024-03-24T11:49:30.868328+0000","last_active":"2024-03-24T11:49:57.142147+0000","last_peered":"2024-03-24T11:49:57.142147+0000","last_clean":"2024-03-24T11:49:57.142147+0000","last_became_active":"2024-03-24T11:49:30.867467+0000","last_became_peered":"2024-03-24T11:49:30.867467+0000","last_unstale":"2024-03-24T11:49:57.142147+0000","last_undegraded":"2024-03-24T11:49:57.142147+0000","last_fullsized":"2024-03-24T11:49:57.142147+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:18:46.136182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139873+0000","last_change":"2024-03-24T11:49:26.836326+0000","last_active":"2024-03-24T11:49:57.139873+0000","last_peered":"2024-03-24T11:49:57.139873+0000","last_clean":"2024-03-24T11:49:57.139873+0000","last_became_active":"2024-03-24T11:49:26.836005+0000","last_became_peered":"2024-03-24T11:49:26.836005+0000","last_unstale":"2024-03-24T11:49:57.139873+0000","last_undegraded":"2024-03-24T11:49:57.139873+0000","last_fullsized":"2024-03-24T11:49:57.139873+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:49:41.808088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139161+0000","last_change":"2024-03-24T11:49:28.849954+0000","last_active":"2024-03-24T11:49:57.139161+0000","last_peered":"2024-03-24T11:49:57.139161+0000","last_clean":"2024-03-24T11:49:57.139161+0000","last_became_active":"2024-03-24T11:49:28.849663+0000","last_became_peered":"2024-03-24T11:49:28.849663+0000","last_unstale":"2024-03-24T11:49:57.139161+0000","last_undegraded":"2024-03-24T11:49:57.139161+0000","last_fullsized":"2024-03-24T11:49:57.139161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:49:58.706757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139957+0000","last_change":"2024-03-24T11:49:32.882079+0000","last_active":"2024-03-24T11:49:57.139957+0000","last_peered":"2024-03-24T11:49:57.139957+0000","last_clean":"2024-03-24T11:49:57.139957+0000","last_became_active":"2024-03-24T11:49:32.881805+0000","last_became_peered":"2024-03-24T11:49:32.881805+0000","last_unstale":"2024-03-24T11:49:57.139957+0000","last_undegraded":"2024-03-24T11:49:57.139957+0000","last_fullsized":"2024-03-24T11:49:57.139957+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:26:41.311574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139473+0000","last_change":"2024-03-24T11:49:26.837054+0000","last_active":"2024-03-24T11:49:57.139473+0000","last_peered":"2024-03-24T11:49:57.139473+0000","last_clean":"2024-03-24T11:49:57.139473+0000","last_became_active":"2024-03-24T11:49:26.836652+0000","last_became_peered":"2024-03-24T11:49:26.836652+0000","last_unstale":"2024-03-24T11:49:57.139473+0000","last_undegraded":"2024-03-24T11:49:57.139473+0000","last_fullsized":"2024-03-24T11:49:57.139473+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:41:18.299556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140346+0000","last_change":"2024-03-24T11:49:28.850275+0000","last_active":"2024-03-24T11:49:57.140346+0000","last_peered":"2024-03-24T11:49:57.140346+0000","last_clean":"2024-03-24T11:49:57.140346+0000","last_became_active":"2024-03-24T11:49:28.849729+0000","last_became_peered":"2024-03-24T11:49:28.849729+0000","last_unstale":"2024-03-24T11:49:57.140346+0000","last_undegraded":"2024-03-24T11:49:57.140346+0000","last_fullsized":"2024-03-24T11:49:57.140346+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:22:50.537818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140245+0000","last_change":"2024-03-24T11:49:30.860562+0000","last_active":"2024-03-24T11:49:57.140245+0000","last_peered":"2024-03-24T11:49:57.140245+0000","last_clean":"2024-03-24T11:49:57.140245+0000","last_became_active":"2024-03-24T11:49:30.859689+0000","last_became_peered":"2024-03-24T11:49:30.859689+0000","last_unstale":"2024-03-24T11:49:57.140245+0000","last_undegraded":"2024-03-24T11:49:57.140245+0000","last_fullsized":"2024-03-24T11:49:57.140245+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:34:11.801292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139577+0000","last_change":"2024-03-24T11:49:32.883684+0000","last_active":"2024-03-24T11:49:57.139577+0000","last_peered":"2024-03-24T11:49:57.139577+0000","last_clean":"2024-03-24T11:49:57.139577+0000","last_became_active":"2024-03-24T11:49:32.883301+0000","last_became_peered":"2024-03-24T11:49:32.883301+0000","last_unstale":"2024-03-24T11:49:57.139577+0000","last_undegraded":"2024-03-24T11:49:57.139577+0000","last_fullsized":"2024-03-24T11:49:57.139577+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:10:43.995441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":35,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140648+0000","last_change":"2024-03-24T11:49:30.865157+0000","last_active":"2024-03-24T11:49:57.140648+0000","last_peered":"2024-03-24T11:49:57.140648+0000","last_clean":"2024-03-24T11:49:57.140648+0000","last_became_active":"2024-03-24T11:49:30.864685+0000","last_became_peered":"2024-03-24T11:49:30.864685+0000","last_unstale":"2024-03-24T11:49:57.140648+0000","last_undegraded":"2024-03-24T11:49:57.140648+0000","last_fullsized":"2024-03-24T11:49:57.140648+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:52:04.753824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139764+0000","last_change":"2024-03-24T11:49:26.832963+0000","last_active":"2024-03-24T11:49:57.139764+0000","last_peered":"2024-03-24T11:49:57.139764+0000","last_clean":"2024-03-24T11:49:57.139764+0000","last_became_active":"2024-03-24T11:49:26.832354+0000","last_became_peered":"2024-03-24T11:49:26.832354+0000","last_unstale":"2024-03-24T11:49:57.139764+0000","last_undegraded":"2024-03-24T11:49:57.139764+0000","last_fullsized":"2024-03-24T11:49:57.139764+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:56:17.366171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'19","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139368+0000","last_change":"2024-03-24T11:49:28.848678+0000","last_active":"2024-03-24T11:49:57.139368+0000","last_peered":"2024-03-24T11:49:57.139368+0000","last_clean":"2024-03-24T11:49:57.139368+0000","last_became_active":"2024-03-24T11:49:28.847873+0000","last_became_peered":"2024-03-24T11:49:28.847873+0000","last_unstale":"2024-03-24T11:49:57.139368+0000","last_undegraded":"2024-03-24T11:49:57.139368+0000","last_fullsized":"2024-03-24T11:49:57.139368+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:17:48.243747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139105+0000","last_change":"2024-03-24T11:49:32.880703+0000","last_active":"2024-03-24T11:49:57.139105+0000","last_peered":"2024-03-24T11:49:57.139105+0000","last_clean":"2024-03-24T11:49:57.139105+0000","last_became_active":"2024-03-24T11:49:32.880435+0000","last_became_peered":"2024-03-24T11:49:32.880435+0000","last_unstale":"2024-03-24T11:49:57.139105+0000","last_undegraded":"2024-03-24T11:49:57.139105+0000","last_fullsized":"2024-03-24T11:49:57.139105+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:45:08.074039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140411+0000","last_change":"2024-03-24T11:49:26.831463+0000","last_active":"2024-03-24T11:49:57.140411+0000","last_peered":"2024-03-24T11:49:57.140411+0000","last_clean":"2024-03-24T11:49:57.140411+0000","last_became_active":"2024-03-24T11:49:26.831106+0000","last_became_peered":"2024-03-24T11:49:26.831106+0000","last_unstale":"2024-03-24T11:49:57.140411+0000","last_undegraded":"2024-03-24T11:49:57.140411+0000","last_fullsized":"2024-03-24T11:49:57.140411+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:04.719567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141149+0000","last_change":"2024-03-24T11:49:28.850007+0000","last_active":"2024-03-24T11:49:57.141149+0000","last_peered":"2024-03-24T11:49:57.141149+0000","last_clean":"2024-03-24T11:49:57.141149+0000","last_became_active":"2024-03-24T11:49:28.848956+0000","last_became_peered":"2024-03-24T11:49:28.848956+0000","last_unstale":"2024-03-24T11:49:57.141149+0000","last_undegraded":"2024-03-24T11:49:57.141149+0000","last_fullsized":"2024-03-24T11:49:57.141149+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:32.532840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139195+0000","last_change":"2024-03-24T11:49:30.860370+0000","last_active":"2024-03-24T11:49:57.139195+0000","last_peered":"2024-03-24T11:49:57.139195+0000","last_clean":"2024-03-24T11:49:57.139195+0000","last_became_active":"2024-03-24T11:49:30.859522+0000","last_became_peered":"2024-03-24T11:49:30.859522+0000","last_unstale":"2024-03-24T11:49:57.139195+0000","last_undegraded":"2024-03-24T11:49:57.139195+0000","last_fullsized":"2024-03-24T11:49:57.139195+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:48:35.511349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139380+0000","last_change":"2024-03-24T11:49:32.881195+0000","last_active":"2024-03-24T11:49:57.139380+0000","last_peered":"2024-03-24T11:49:57.139380+0000","last_clean":"2024-03-24T11:49:57.139380+0000","last_became_active":"2024-03-24T11:49:32.880785+0000","last_became_peered":"2024-03-24T11:49:32.880785+0000","last_unstale":"2024-03-24T11:49:57.139380+0000","last_undegraded":"2024-03-24T11:49:57.139380+0000","last_fullsized":"2024-03-24T11:49:57.139380+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:23.321147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141195+0000","last_change":"2024-03-24T11:49:26.836789+0000","last_active":"2024-03-24T11:49:57.141195+0000","last_peered":"2024-03-24T11:49:57.141195+0000","last_clean":"2024-03-24T11:49:57.141195+0000","last_became_active":"2024-03-24T11:49:26.835223+0000","last_became_peered":"2024-03-24T11:49:26.835223+0000","last_unstale":"2024-03-24T11:49:57.141195+0000","last_undegraded":"2024-03-24T11:49:57.141195+0000","last_fullsized":"2024-03-24T11:49:57.141195+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:04:45.325722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140734+0000","last_change":"2024-03-24T11:49:28.850665+0000","last_active":"2024-03-24T11:49:57.140734+0000","last_peered":"2024-03-24T11:49:57.140734+0000","last_clean":"2024-03-24T11:49:57.140734+0000","last_became_active":"2024-03-24T11:49:28.850403+0000","last_became_peered":"2024-03-24T11:49:28.850403+0000","last_unstale":"2024-03-24T11:49:57.140734+0000","last_undegraded":"2024-03-24T11:49:57.140734+0000","last_fullsized":"2024-03-24T11:49:57.140734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:47.518866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139644+0000","last_change":"2024-03-24T11:49:30.867113+0000","last_active":"2024-03-24T11:49:57.139644+0000","last_peered":"2024-03-24T11:49:57.139644+0000","last_clean":"2024-03-24T11:49:57.139644+0000","last_became_active":"2024-03-24T11:49:30.866777+0000","last_became_peered":"2024-03-24T11:49:30.866777+0000","last_unstale":"2024-03-24T11:49:57.139644+0000","last_undegraded":"2024-03-24T11:49:57.139644+0000","last_fullsized":"2024-03-24T11:49:57.139644+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:48:35.183573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138679+0000","last_change":"2024-03-24T11:49:32.881734+0000","last_active":"2024-03-24T11:49:57.138679+0000","last_peered":"2024-03-24T11:49:57.138679+0000","last_clean":"2024-03-24T11:49:57.138679+0000","last_became_active":"2024-03-24T11:49:32.880387+0000","last_became_peered":"2024-03-24T11:49:32.880387+0000","last_unstale":"2024-03-24T11:49:57.138679+0000","last_undegraded":"2024-03-24T11:49:57.138679+0000","last_fullsized":"2024-03-24T11:49:57.138679+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:06.732552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141995+0000","last_change":"2024-03-24T11:49:26.836399+0000","last_active":"2024-03-24T11:49:57.141995+0000","last_peered":"2024-03-24T11:49:57.141995+0000","last_clean":"2024-03-24T11:49:57.141995+0000","last_became_active":"2024-03-24T11:49:26.836121+0000","last_became_peered":"2024-03-24T11:49:26.836121+0000","last_unstale":"2024-03-24T11:49:57.141995+0000","last_undegraded":"2024-03-24T11:49:57.141995+0000","last_fullsized":"2024-03-24T11:49:57.141995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:01:16.583166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141203+0000","last_change":"2024-03-24T11:49:28.849143+0000","last_active":"2024-03-24T11:49:57.141203+0000","last_peered":"2024-03-24T11:49:57.141203+0000","last_clean":"2024-03-24T11:49:57.141203+0000","last_became_active":"2024-03-24T11:49:28.848715+0000","last_became_peered":"2024-03-24T11:49:28.848715+0000","last_unstale":"2024-03-24T11:49:57.141203+0000","last_undegraded":"2024-03-24T11:49:57.141203+0000","last_fullsized":"2024-03-24T11:49:57.141203+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:26:44.048534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139372+0000","last_change":"2024-03-24T11:49:30.865169+0000","last_active":"2024-03-24T11:49:57.139372+0000","last_peered":"2024-03-24T11:49:57.139372+0000","last_clean":"2024-03-24T11:49:57.139372+0000","last_became_active":"2024-03-24T11:49:30.864773+0000","last_became_peered":"2024-03-24T11:49:30.864773+0000","last_unstale":"2024-03-24T11:49:57.139372+0000","last_undegraded":"2024-03-24T11:49:57.139372+0000","last_fullsized":"2024-03-24T11:49:57.139372+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:28:10.860288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411340,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708840,"statfs":{"total":95990841344,"available":95957852160,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738390,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708840,"statfs":{"total":95990841344,"available":95957852160,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708840,"statfs":{"total":95990841344,"available":95957852160,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:50:08.781 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:50:09.020 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:09.037 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:08 smithi130 ceph-mon[25795]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:50:09.037 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:08 smithi130 ceph-mon[25795]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:50:09.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:08 smithi174 ceph-mon[30177]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:50:09.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:08 smithi174 ceph-mon[30177]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:50:09.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:08 smithi189 ceph-mon[30629]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:50:09.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:08 smithi189 ceph-mon[30629]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:50:11.287 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:10 smithi130 ceph-mon[25795]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:50:11.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:10 smithi174 ceph-mon[30177]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:50:11.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:10 smithi189 ceph-mon[30629]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:50:12.199 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:50:12.201 INFO:teuthology.orchestra.run.smithi130.stderr:dumped all 2024-03-24T11:50:13.227 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:12 smithi130 ceph-mon[25795]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:50:13.227 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:12 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:13.259 INFO:teuthology.orchestra.run.smithi130.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-24T11:50:11.151542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96648,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126520,"statfs":{"total":287972524032,"available":287873556480,"internally_reserved":0,"allocated":3182592,"data_stored":2109504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002185"},"pg_stats":[{"pgid":"5.18","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138519+0000","last_change":"2024-03-24T11:49:32.882079+0000","last_active":"2024-03-24T11:49:57.138519+0000","last_peered":"2024-03-24T11:49:57.138519+0000","last_clean":"2024-03-24T11:49:57.138519+0000","last_became_active":"2024-03-24T11:49:32.881514+0000","last_became_peered":"2024-03-24T11:49:32.881514+0000","last_unstale":"2024-03-24T11:49:57.138519+0000","last_undegraded":"2024-03-24T11:49:57.138519+0000","last_fullsized":"2024-03-24T11:49:57.138519+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:58:33.328337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138556+0000","last_change":"2024-03-24T11:49:30.865209+0000","last_active":"2024-03-24T11:49:57.138556+0000","last_peered":"2024-03-24T11:49:57.138556+0000","last_clean":"2024-03-24T11:49:57.138556+0000","last_became_active":"2024-03-24T11:49:30.864715+0000","last_became_peered":"2024-03-24T11:49:30.864715+0000","last_unstale":"2024-03-24T11:49:57.138556+0000","last_undegraded":"2024-03-24T11:49:57.138556+0000","last_fullsized":"2024-03-24T11:49:57.138556+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:21:35.244233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.142355+0000","last_change":"2024-03-24T11:49:26.837779+0000","last_active":"2024-03-24T11:49:57.142355+0000","last_peered":"2024-03-24T11:49:57.142355+0000","last_clean":"2024-03-24T11:49:57.142355+0000","last_became_active":"2024-03-24T11:49:26.837496+0000","last_became_peered":"2024-03-24T11:49:26.837496+0000","last_unstale":"2024-03-24T11:49:57.142355+0000","last_undegraded":"2024-03-24T11:49:57.142355+0000","last_fullsized":"2024-03-24T11:49:57.142355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:03:39.447745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.142014+0000","last_change":"2024-03-24T11:49:28.852718+0000","last_active":"2024-03-24T11:49:57.142014+0000","last_peered":"2024-03-24T11:49:57.142014+0000","last_clean":"2024-03-24T11:49:57.142014+0000","last_became_active":"2024-03-24T11:49:28.852394+0000","last_became_peered":"2024-03-24T11:49:28.852394+0000","last_unstale":"2024-03-24T11:49:57.142014+0000","last_undegraded":"2024-03-24T11:49:57.142014+0000","last_fullsized":"2024-03-24T11:49:57.142014+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:59:11.078342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139744+0000","last_change":"2024-03-24T11:49:32.878867+0000","last_active":"2024-03-24T11:49:57.139744+0000","last_peered":"2024-03-24T11:49:57.139744+0000","last_clean":"2024-03-24T11:49:57.139744+0000","last_became_active":"2024-03-24T11:49:32.878508+0000","last_became_peered":"2024-03-24T11:49:32.878508+0000","last_unstale":"2024-03-24T11:49:57.139744+0000","last_undegraded":"2024-03-24T11:49:57.139744+0000","last_fullsized":"2024-03-24T11:49:57.139744+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:46:43.266909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138975+0000","last_change":"2024-03-24T11:49:30.866542+0000","last_active":"2024-03-24T11:49:57.138975+0000","last_peered":"2024-03-24T11:49:57.138975+0000","last_clean":"2024-03-24T11:49:57.138975+0000","last_became_active":"2024-03-24T11:49:30.865898+0000","last_became_peered":"2024-03-24T11:49:30.865898+0000","last_unstale":"2024-03-24T11:49:57.138975+0000","last_undegraded":"2024-03-24T11:49:57.138975+0000","last_fullsized":"2024-03-24T11:49:57.138975+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:57.781307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141859+0000","last_change":"2024-03-24T11:49:26.837746+0000","last_active":"2024-03-24T11:49:57.141859+0000","last_peered":"2024-03-24T11:49:57.141859+0000","last_clean":"2024-03-24T11:49:57.141859+0000","last_became_active":"2024-03-24T11:49:26.837266+0000","last_became_peered":"2024-03-24T11:49:26.837266+0000","last_unstale":"2024-03-24T11:49:57.141859+0000","last_undegraded":"2024-03-24T11:49:57.141859+0000","last_fullsized":"2024-03-24T11:49:57.141859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:06.966471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.142221+0000","last_change":"2024-03-24T11:49:28.851532+0000","last_active":"2024-03-24T11:49:57.142221+0000","last_peered":"2024-03-24T11:49:57.142221+0000","last_clean":"2024-03-24T11:49:57.142221+0000","last_became_active":"2024-03-24T11:49:28.850989+0000","last_became_peered":"2024-03-24T11:49:28.850989+0000","last_unstale":"2024-03-24T11:49:57.142221+0000","last_undegraded":"2024-03-24T11:49:57.142221+0000","last_fullsized":"2024-03-24T11:49:57.142221+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:05:19.369840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138707+0000","last_change":"2024-03-24T11:49:32.881307+0000","last_active":"2024-03-24T11:49:57.138707+0000","last_peered":"2024-03-24T11:49:57.138707+0000","last_clean":"2024-03-24T11:49:57.138707+0000","last_became_active":"2024-03-24T11:49:32.881022+0000","last_became_peered":"2024-03-24T11:49:32.881022+0000","last_unstale":"2024-03-24T11:49:57.138707+0000","last_undegraded":"2024-03-24T11:49:57.138707+0000","last_fullsized":"2024-03-24T11:49:57.138707+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:57:28.804952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139337+0000","last_change":"2024-03-24T11:49:30.860570+0000","last_active":"2024-03-24T11:49:57.139337+0000","last_peered":"2024-03-24T11:49:57.139337+0000","last_clean":"2024-03-24T11:49:57.139337+0000","last_became_active":"2024-03-24T11:49:30.859647+0000","last_became_peered":"2024-03-24T11:49:30.859647+0000","last_unstale":"2024-03-24T11:49:57.139337+0000","last_undegraded":"2024-03-24T11:49:57.139337+0000","last_fullsized":"2024-03-24T11:49:57.139337+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:25.166269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141588+0000","last_change":"2024-03-24T11:49:26.835507+0000","last_active":"2024-03-24T11:49:57.141588+0000","last_peered":"2024-03-24T11:49:57.141588+0000","last_clean":"2024-03-24T11:49:57.141588+0000","last_became_active":"2024-03-24T11:49:26.834725+0000","last_became_peered":"2024-03-24T11:49:26.834725+0000","last_unstale":"2024-03-24T11:49:57.141588+0000","last_undegraded":"2024-03-24T11:49:57.141588+0000","last_fullsized":"2024-03-24T11:49:57.141588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:27.310999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140777+0000","last_change":"2024-03-24T11:49:28.850310+0000","last_active":"2024-03-24T11:49:57.140777+0000","last_peered":"2024-03-24T11:49:57.140777+0000","last_clean":"2024-03-24T11:49:57.140777+0000","last_became_active":"2024-03-24T11:49:28.848953+0000","last_became_peered":"2024-03-24T11:49:28.848953+0000","last_unstale":"2024-03-24T11:49:57.140777+0000","last_undegraded":"2024-03-24T11:49:57.140777+0000","last_fullsized":"2024-03-24T11:49:57.140777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:33:13.084663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139120+0000","last_change":"2024-03-24T11:49:32.881821+0000","last_active":"2024-03-24T11:49:57.139120+0000","last_peered":"2024-03-24T11:49:57.139120+0000","last_clean":"2024-03-24T11:49:57.139120+0000","last_became_active":"2024-03-24T11:49:32.881276+0000","last_became_peered":"2024-03-24T11:49:32.881276+0000","last_unstale":"2024-03-24T11:49:57.139120+0000","last_undegraded":"2024-03-24T11:49:57.139120+0000","last_fullsized":"2024-03-24T11:49:57.139120+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:19:06.054454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138938+0000","last_change":"2024-03-24T11:49:30.864629+0000","last_active":"2024-03-24T11:49:57.138938+0000","last_peered":"2024-03-24T11:49:57.138938+0000","last_clean":"2024-03-24T11:49:57.138938+0000","last_became_active":"2024-03-24T11:49:30.864062+0000","last_became_peered":"2024-03-24T11:49:30.864062+0000","last_unstale":"2024-03-24T11:49:57.138938+0000","last_undegraded":"2024-03-24T11:49:57.138938+0000","last_fullsized":"2024-03-24T11:49:57.138938+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:22:47.412210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140304+0000","last_change":"2024-03-24T11:49:26.837098+0000","last_active":"2024-03-24T11:49:57.140304+0000","last_peered":"2024-03-24T11:49:57.140304+0000","last_clean":"2024-03-24T11:49:57.140304+0000","last_became_active":"2024-03-24T11:49:26.836737+0000","last_became_peered":"2024-03-24T11:49:26.836737+0000","last_unstale":"2024-03-24T11:49:57.140304+0000","last_undegraded":"2024-03-24T11:49:57.140304+0000","last_fullsized":"2024-03-24T11:49:57.140304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:53:38.719175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141486+0000","last_change":"2024-03-24T11:49:28.850595+0000","last_active":"2024-03-24T11:49:57.141486+0000","last_peered":"2024-03-24T11:49:57.141486+0000","last_clean":"2024-03-24T11:49:57.141486+0000","last_became_active":"2024-03-24T11:49:28.850122+0000","last_became_peered":"2024-03-24T11:49:28.850122+0000","last_unstale":"2024-03-24T11:49:57.141486+0000","last_undegraded":"2024-03-24T11:49:57.141486+0000","last_fullsized":"2024-03-24T11:49:57.141486+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:41:23.234445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140244+0000","last_change":"2024-03-24T11:49:32.883293+0000","last_active":"2024-03-24T11:49:57.140244+0000","last_peered":"2024-03-24T11:49:57.140244+0000","last_clean":"2024-03-24T11:49:57.140244+0000","last_became_active":"2024-03-24T11:49:32.882892+0000","last_became_peered":"2024-03-24T11:49:32.882892+0000","last_unstale":"2024-03-24T11:49:57.140244+0000","last_undegraded":"2024-03-24T11:49:57.140244+0000","last_fullsized":"2024-03-24T11:49:57.140244+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:59.428492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140064+0000","last_change":"2024-03-24T11:49:30.864544+0000","last_active":"2024-03-24T11:49:57.140064+0000","last_peered":"2024-03-24T11:49:57.140064+0000","last_clean":"2024-03-24T11:49:57.140064+0000","last_became_active":"2024-03-24T11:49:30.864240+0000","last_became_peered":"2024-03-24T11:49:30.864240+0000","last_unstale":"2024-03-24T11:49:57.140064+0000","last_undegraded":"2024-03-24T11:49:57.140064+0000","last_fullsized":"2024-03-24T11:49:57.140064+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:12:02.582475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140004+0000","last_change":"2024-03-24T11:49:26.834671+0000","last_active":"2024-03-24T11:49:57.140004+0000","last_peered":"2024-03-24T11:49:57.140004+0000","last_clean":"2024-03-24T11:49:57.140004+0000","last_became_active":"2024-03-24T11:49:26.834255+0000","last_became_peered":"2024-03-24T11:49:26.834255+0000","last_unstale":"2024-03-24T11:49:57.140004+0000","last_undegraded":"2024-03-24T11:49:57.140004+0000","last_fullsized":"2024-03-24T11:49:57.140004+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:19:36.215970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139543+0000","last_change":"2024-03-24T11:49:28.851080+0000","last_active":"2024-03-24T11:49:57.139543+0000","last_peered":"2024-03-24T11:49:57.139543+0000","last_clean":"2024-03-24T11:49:57.139543+0000","last_became_active":"2024-03-24T11:49:28.850705+0000","last_became_peered":"2024-03-24T11:49:28.850705+0000","last_unstale":"2024-03-24T11:49:57.139543+0000","last_undegraded":"2024-03-24T11:49:57.139543+0000","last_fullsized":"2024-03-24T11:49:57.139543+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:48.230899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141608+0000","last_change":"2024-03-24T11:49:32.885182+0000","last_active":"2024-03-24T11:49:57.141608+0000","last_peered":"2024-03-24T11:49:57.141608+0000","last_clean":"2024-03-24T11:49:57.141608+0000","last_became_active":"2024-03-24T11:49:32.884860+0000","last_became_peered":"2024-03-24T11:49:32.884860+0000","last_unstale":"2024-03-24T11:49:57.141608+0000","last_undegraded":"2024-03-24T11:49:57.141608+0000","last_fullsized":"2024-03-24T11:49:57.141608+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:01:48.866841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140185+0000","last_change":"2024-03-24T11:49:30.862551+0000","last_active":"2024-03-24T11:49:57.140185+0000","last_peered":"2024-03-24T11:49:57.140185+0000","last_clean":"2024-03-24T11:49:57.140185+0000","last_became_active":"2024-03-24T11:49:30.861646+0000","last_became_peered":"2024-03-24T11:49:30.861646+0000","last_unstale":"2024-03-24T11:49:57.140185+0000","last_undegraded":"2024-03-24T11:49:57.140185+0000","last_fullsized":"2024-03-24T11:49:57.140185+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:08:55.361053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139615+0000","last_change":"2024-03-24T11:49:26.833093+0000","last_active":"2024-03-24T11:49:57.139615+0000","last_peered":"2024-03-24T11:49:57.139615+0000","last_clean":"2024-03-24T11:49:57.139615+0000","last_became_active":"2024-03-24T11:49:26.832600+0000","last_became_peered":"2024-03-24T11:49:26.832600+0000","last_unstale":"2024-03-24T11:49:57.139615+0000","last_undegraded":"2024-03-24T11:49:57.139615+0000","last_fullsized":"2024-03-24T11:49:57.139615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:02.898944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139627+0000","last_change":"2024-03-24T11:49:28.850890+0000","last_active":"2024-03-24T11:49:57.139627+0000","last_peered":"2024-03-24T11:49:57.139627+0000","last_clean":"2024-03-24T11:49:57.139627+0000","last_became_active":"2024-03-24T11:49:28.849717+0000","last_became_peered":"2024-03-24T11:49:28.849717+0000","last_unstale":"2024-03-24T11:49:57.139627+0000","last_undegraded":"2024-03-24T11:49:57.139627+0000","last_fullsized":"2024-03-24T11:49:57.139627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:09:15.966208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140631+0000","last_change":"2024-03-24T11:49:32.884470+0000","last_active":"2024-03-24T11:49:57.140631+0000","last_peered":"2024-03-24T11:49:57.140631+0000","last_clean":"2024-03-24T11:49:57.140631+0000","last_became_active":"2024-03-24T11:49:32.884036+0000","last_became_peered":"2024-03-24T11:49:32.884036+0000","last_unstale":"2024-03-24T11:49:57.140631+0000","last_undegraded":"2024-03-24T11:49:57.140631+0000","last_fullsized":"2024-03-24T11:49:57.140631+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:43:08.044397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:50:01.879585+0000","last_change":"2024-03-24T11:49:30.859156+0000","last_active":"2024-03-24T11:50:01.879585+0000","last_peered":"2024-03-24T11:50:01.879585+0000","last_clean":"2024-03-24T11:50:01.879585+0000","last_became_active":"2024-03-24T11:49:30.858799+0000","last_became_peered":"2024-03-24T11:49:30.858799+0000","last_unstale":"2024-03-24T11:50:01.879585+0000","last_undegraded":"2024-03-24T11:50:01.879585+0000","last_fullsized":"2024-03-24T11:50:01.879585+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:17:30.150902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140292+0000","last_change":"2024-03-24T11:49:26.837563+0000","last_active":"2024-03-24T11:49:57.140292+0000","last_peered":"2024-03-24T11:49:57.140292+0000","last_clean":"2024-03-24T11:49:57.140292+0000","last_became_active":"2024-03-24T11:49:26.837241+0000","last_became_peered":"2024-03-24T11:49:26.837241+0000","last_unstale":"2024-03-24T11:49:57.140292+0000","last_undegraded":"2024-03-24T11:49:57.140292+0000","last_fullsized":"2024-03-24T11:49:57.140292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:51:21.736037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139285+0000","last_change":"2024-03-24T11:49:28.854103+0000","last_active":"2024-03-24T11:49:57.139285+0000","last_peered":"2024-03-24T11:49:57.139285+0000","last_clean":"2024-03-24T11:49:57.139285+0000","last_became_active":"2024-03-24T11:49:28.853327+0000","last_became_peered":"2024-03-24T11:49:28.853327+0000","last_unstale":"2024-03-24T11:49:57.139285+0000","last_undegraded":"2024-03-24T11:49:57.139285+0000","last_fullsized":"2024-03-24T11:49:57.139285+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:57:33.372728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140310+0000","last_change":"2024-03-24T11:49:32.882956+0000","last_active":"2024-03-24T11:49:57.140310+0000","last_peered":"2024-03-24T11:49:57.140310+0000","last_clean":"2024-03-24T11:49:57.140310+0000","last_became_active":"2024-03-24T11:49:32.882540+0000","last_became_peered":"2024-03-24T11:49:32.882540+0000","last_unstale":"2024-03-24T11:49:57.140310+0000","last_undegraded":"2024-03-24T11:49:57.140310+0000","last_fullsized":"2024-03-24T11:49:57.140310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:46:27.366070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140341+0000","last_change":"2024-03-24T11:49:30.868423+0000","last_active":"2024-03-24T11:49:57.140341+0000","last_peered":"2024-03-24T11:49:57.140341+0000","last_clean":"2024-03-24T11:49:57.140341+0000","last_became_active":"2024-03-24T11:49:30.867736+0000","last_became_peered":"2024-03-24T11:49:30.867736+0000","last_unstale":"2024-03-24T11:49:57.140341+0000","last_undegraded":"2024-03-24T11:49:57.140341+0000","last_fullsized":"2024-03-24T11:49:57.140341+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:00:56.171282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138797+0000","last_change":"2024-03-24T11:49:26.837803+0000","last_active":"2024-03-24T11:49:57.138797+0000","last_peered":"2024-03-24T11:49:57.138797+0000","last_clean":"2024-03-24T11:49:57.138797+0000","last_became_active":"2024-03-24T11:49:26.837230+0000","last_became_peered":"2024-03-24T11:49:26.837230+0000","last_unstale":"2024-03-24T11:49:57.138797+0000","last_undegraded":"2024-03-24T11:49:57.138797+0000","last_fullsized":"2024-03-24T11:49:57.138797+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:20:38.306158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139593+0000","last_change":"2024-03-24T11:49:28.850022+0000","last_active":"2024-03-24T11:49:57.139593+0000","last_peered":"2024-03-24T11:49:57.139593+0000","last_clean":"2024-03-24T11:49:57.139593+0000","last_became_active":"2024-03-24T11:49:28.849624+0000","last_became_peered":"2024-03-24T11:49:28.849624+0000","last_unstale":"2024-03-24T11:49:57.139593+0000","last_undegraded":"2024-03-24T11:49:57.139593+0000","last_fullsized":"2024-03-24T11:49:57.139593+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:19.627510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139233+0000","last_change":"2024-03-24T11:49:32.883178+0000","last_active":"2024-03-24T11:49:57.139233+0000","last_peered":"2024-03-24T11:49:57.139233+0000","last_clean":"2024-03-24T11:49:57.139233+0000","last_became_active":"2024-03-24T11:49:32.882455+0000","last_became_peered":"2024-03-24T11:49:32.882455+0000","last_unstale":"2024-03-24T11:49:57.139233+0000","last_undegraded":"2024-03-24T11:49:57.139233+0000","last_fullsized":"2024-03-24T11:49:57.139233+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:49:02.276165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139236+0000","last_change":"2024-03-24T11:49:30.869002+0000","last_active":"2024-03-24T11:49:57.139236+0000","last_peered":"2024-03-24T11:49:57.139236+0000","last_clean":"2024-03-24T11:49:57.139236+0000","last_became_active":"2024-03-24T11:49:30.868697+0000","last_became_peered":"2024-03-24T11:49:30.868697+0000","last_unstale":"2024-03-24T11:49:57.139236+0000","last_undegraded":"2024-03-24T11:49:57.139236+0000","last_fullsized":"2024-03-24T11:49:57.139236+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:19.305750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141461+0000","last_change":"2024-03-24T11:49:26.831136+0000","last_active":"2024-03-24T11:49:57.141461+0000","last_peered":"2024-03-24T11:49:57.141461+0000","last_clean":"2024-03-24T11:49:57.141461+0000","last_became_active":"2024-03-24T11:49:26.830219+0000","last_became_peered":"2024-03-24T11:49:26.830219+0000","last_unstale":"2024-03-24T11:49:57.141461+0000","last_undegraded":"2024-03-24T11:49:57.141461+0000","last_fullsized":"2024-03-24T11:49:57.141461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:38:31.342476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141113+0000","last_change":"2024-03-24T11:49:28.850101+0000","last_active":"2024-03-24T11:49:57.141113+0000","last_peered":"2024-03-24T11:49:57.141113+0000","last_clean":"2024-03-24T11:49:57.141113+0000","last_became_active":"2024-03-24T11:49:28.849021+0000","last_became_peered":"2024-03-24T11:49:28.849021+0000","last_unstale":"2024-03-24T11:49:57.141113+0000","last_undegraded":"2024-03-24T11:49:57.141113+0000","last_fullsized":"2024-03-24T11:49:57.141113+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:42:21.864632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138952+0000","last_change":"2024-03-24T11:49:32.880694+0000","last_active":"2024-03-24T11:49:57.138952+0000","last_peered":"2024-03-24T11:49:57.138952+0000","last_clean":"2024-03-24T11:49:57.138952+0000","last_became_active":"2024-03-24T11:49:32.880288+0000","last_became_peered":"2024-03-24T11:49:32.880288+0000","last_unstale":"2024-03-24T11:49:57.138952+0000","last_undegraded":"2024-03-24T11:49:57.138952+0000","last_fullsized":"2024-03-24T11:49:57.138952+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:12:03.173738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139574+0000","last_change":"2024-03-24T11:49:30.867703+0000","last_active":"2024-03-24T11:49:57.139574+0000","last_peered":"2024-03-24T11:49:57.139574+0000","last_clean":"2024-03-24T11:49:57.139574+0000","last_became_active":"2024-03-24T11:49:30.866776+0000","last_became_peered":"2024-03-24T11:49:30.866776+0000","last_unstale":"2024-03-24T11:49:57.139574+0000","last_undegraded":"2024-03-24T11:49:57.139574+0000","last_fullsized":"2024-03-24T11:49:57.139574+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:11:18.232452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140853+0000","last_change":"2024-03-24T11:49:26.831215+0000","last_active":"2024-03-24T11:49:57.140853+0000","last_peered":"2024-03-24T11:49:57.140853+0000","last_clean":"2024-03-24T11:49:57.140853+0000","last_became_active":"2024-03-24T11:49:26.830189+0000","last_became_peered":"2024-03-24T11:49:26.830189+0000","last_unstale":"2024-03-24T11:49:57.140853+0000","last_undegraded":"2024-03-24T11:49:57.140853+0000","last_fullsized":"2024-03-24T11:49:57.140853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:54:01.277105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.143323+0000","last_change":"2024-03-24T11:49:28.853115+0000","last_active":"2024-03-24T11:49:57.143323+0000","last_peered":"2024-03-24T11:49:57.143323+0000","last_clean":"2024-03-24T11:49:57.143323+0000","last_became_active":"2024-03-24T11:49:28.852529+0000","last_became_peered":"2024-03-24T11:49:28.852529+0000","last_unstale":"2024-03-24T11:49:57.143323+0000","last_undegraded":"2024-03-24T11:49:57.143323+0000","last_fullsized":"2024-03-24T11:49:57.143323+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:31:40.725251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138877+0000","last_change":"2024-03-24T11:49:32.880745+0000","last_active":"2024-03-24T11:49:57.138877+0000","last_peered":"2024-03-24T11:49:57.138877+0000","last_clean":"2024-03-24T11:49:57.138877+0000","last_became_active":"2024-03-24T11:49:32.880456+0000","last_became_peered":"2024-03-24T11:49:32.880456+0000","last_unstale":"2024-03-24T11:49:57.138877+0000","last_undegraded":"2024-03-24T11:49:57.138877+0000","last_fullsized":"2024-03-24T11:49:57.138877+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:10.240334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139487+0000","last_change":"2024-03-24T11:49:30.860422+0000","last_active":"2024-03-24T11:49:57.139487+0000","last_peered":"2024-03-24T11:49:57.139487+0000","last_clean":"2024-03-24T11:49:57.139487+0000","last_became_active":"2024-03-24T11:49:30.859575+0000","last_became_peered":"2024-03-24T11:49:30.859575+0000","last_unstale":"2024-03-24T11:49:57.139487+0000","last_undegraded":"2024-03-24T11:49:57.139487+0000","last_fullsized":"2024-03-24T11:49:57.139487+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:49:22.363912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141455+0000","last_change":"2024-03-24T11:49:26.831378+0000","last_active":"2024-03-24T11:49:57.141455+0000","last_peered":"2024-03-24T11:49:57.141455+0000","last_clean":"2024-03-24T11:49:57.141455+0000","last_became_active":"2024-03-24T11:49:26.830841+0000","last_became_peered":"2024-03-24T11:49:26.830841+0000","last_unstale":"2024-03-24T11:49:57.141455+0000","last_undegraded":"2024-03-24T11:49:57.141455+0000","last_fullsized":"2024-03-24T11:49:57.141455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:28:22.944311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141160+0000","last_change":"2024-03-24T11:49:28.848238+0000","last_active":"2024-03-24T11:49:57.141160+0000","last_peered":"2024-03-24T11:49:57.141160+0000","last_clean":"2024-03-24T11:49:57.141160+0000","last_became_active":"2024-03-24T11:49:28.847880+0000","last_became_peered":"2024-03-24T11:49:28.847880+0000","last_unstale":"2024-03-24T11:49:57.141160+0000","last_undegraded":"2024-03-24T11:49:57.141160+0000","last_fullsized":"2024-03-24T11:49:57.141160+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:32:35.434942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138947+0000","last_change":"2024-03-24T11:49:32.881543+0000","last_active":"2024-03-24T11:49:57.138947+0000","last_peered":"2024-03-24T11:49:57.138947+0000","last_clean":"2024-03-24T11:49:57.138947+0000","last_became_active":"2024-03-24T11:49:32.881217+0000","last_became_peered":"2024-03-24T11:49:32.881217+0000","last_unstale":"2024-03-24T11:49:57.138947+0000","last_undegraded":"2024-03-24T11:49:57.138947+0000","last_fullsized":"2024-03-24T11:49:57.138947+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:52:15.052552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139833+0000","last_change":"2024-03-24T11:49:30.867578+0000","last_active":"2024-03-24T11:49:57.139833+0000","last_peered":"2024-03-24T11:49:57.139833+0000","last_clean":"2024-03-24T11:49:57.139833+0000","last_became_active":"2024-03-24T11:49:30.867258+0000","last_became_peered":"2024-03-24T11:49:30.867258+0000","last_unstale":"2024-03-24T11:49:57.139833+0000","last_undegraded":"2024-03-24T11:49:57.139833+0000","last_fullsized":"2024-03-24T11:49:57.139833+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:53.755298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140632+0000","last_change":"2024-03-24T11:49:26.835772+0000","last_active":"2024-03-24T11:49:57.140632+0000","last_peered":"2024-03-24T11:49:57.140632+0000","last_clean":"2024-03-24T11:49:57.140632+0000","last_became_active":"2024-03-24T11:49:26.835348+0000","last_became_peered":"2024-03-24T11:49:26.835348+0000","last_unstale":"2024-03-24T11:49:57.140632+0000","last_undegraded":"2024-03-24T11:49:57.140632+0000","last_fullsized":"2024-03-24T11:49:57.140632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:04:20.417857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141267+0000","last_change":"2024-03-24T11:49:28.851086+0000","last_active":"2024-03-24T11:49:57.141267+0000","last_peered":"2024-03-24T11:49:57.141267+0000","last_clean":"2024-03-24T11:49:57.141267+0000","last_became_active":"2024-03-24T11:49:28.850333+0000","last_became_peered":"2024-03-24T11:49:28.850333+0000","last_unstale":"2024-03-24T11:49:57.141267+0000","last_undegraded":"2024-03-24T11:49:57.141267+0000","last_fullsized":"2024-03-24T11:49:57.141267+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:50:19.711047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139672+0000","last_change":"2024-03-24T11:49:32.885050+0000","last_active":"2024-03-24T11:49:57.139672+0000","last_peered":"2024-03-24T11:49:57.139672+0000","last_clean":"2024-03-24T11:49:57.139672+0000","last_became_active":"2024-03-24T11:49:32.884583+0000","last_became_peered":"2024-03-24T11:49:32.884583+0000","last_unstale":"2024-03-24T11:49:57.139672+0000","last_undegraded":"2024-03-24T11:49:57.139672+0000","last_fullsized":"2024-03-24T11:49:57.139672+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:43.946425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:50:01.879597+0000","last_change":"2024-03-24T11:49:30.862644+0000","last_active":"2024-03-24T11:50:01.879597+0000","last_peered":"2024-03-24T11:50:01.879597+0000","last_clean":"2024-03-24T11:50:01.879597+0000","last_became_active":"2024-03-24T11:49:30.862394+0000","last_became_peered":"2024-03-24T11:49:30.862394+0000","last_unstale":"2024-03-24T11:50:01.879597+0000","last_undegraded":"2024-03-24T11:50:01.879597+0000","last_fullsized":"2024-03-24T11:50:01.879597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:07:42.628384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140026+0000","last_change":"2024-03-24T11:49:26.833292+0000","last_active":"2024-03-24T11:49:57.140026+0000","last_peered":"2024-03-24T11:49:57.140026+0000","last_clean":"2024-03-24T11:49:57.140026+0000","last_became_active":"2024-03-24T11:49:26.832980+0000","last_became_peered":"2024-03-24T11:49:26.832980+0000","last_unstale":"2024-03-24T11:49:57.140026+0000","last_undegraded":"2024-03-24T11:49:57.140026+0000","last_fullsized":"2024-03-24T11:49:57.140026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:54:32.094868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140998+0000","last_change":"2024-03-24T11:49:28.850489+0000","last_active":"2024-03-24T11:49:57.140998+0000","last_peered":"2024-03-24T11:49:57.140998+0000","last_clean":"2024-03-24T11:49:57.140998+0000","last_became_active":"2024-03-24T11:49:28.850160+0000","last_became_peered":"2024-03-24T11:49:28.850160+0000","last_unstale":"2024-03-24T11:49:57.140998+0000","last_undegraded":"2024-03-24T11:49:57.140998+0000","last_fullsized":"2024-03-24T11:49:57.140998+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:39:13.549678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139886+0000","last_change":"2024-03-24T11:49:32.881872+0000","last_active":"2024-03-24T11:49:57.139886+0000","last_peered":"2024-03-24T11:49:57.139886+0000","last_clean":"2024-03-24T11:49:57.139886+0000","last_became_active":"2024-03-24T11:49:32.881375+0000","last_became_peered":"2024-03-24T11:49:32.881375+0000","last_unstale":"2024-03-24T11:49:57.139886+0000","last_undegraded":"2024-03-24T11:49:57.139886+0000","last_fullsized":"2024-03-24T11:49:57.139886+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:46:10.592569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:50:01.879568+0000","last_change":"2024-03-24T11:49:30.864462+0000","last_active":"2024-03-24T11:50:01.879568+0000","last_peered":"2024-03-24T11:50:01.879568+0000","last_clean":"2024-03-24T11:50:01.879568+0000","last_became_active":"2024-03-24T11:49:30.864181+0000","last_became_peered":"2024-03-24T11:49:30.864181+0000","last_unstale":"2024-03-24T11:50:01.879568+0000","last_undegraded":"2024-03-24T11:50:01.879568+0000","last_fullsized":"2024-03-24T11:50:01.879568+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:22:19.375051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139157+0000","last_change":"2024-03-24T11:49:26.834596+0000","last_active":"2024-03-24T11:49:57.139157+0000","last_peered":"2024-03-24T11:49:57.139157+0000","last_clean":"2024-03-24T11:49:57.139157+0000","last_became_active":"2024-03-24T11:49:26.833504+0000","last_became_peered":"2024-03-24T11:49:26.833504+0000","last_unstale":"2024-03-24T11:49:57.139157+0000","last_undegraded":"2024-03-24T11:49:57.139157+0000","last_fullsized":"2024-03-24T11:49:57.139157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:32:19.425691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139963+0000","last_change":"2024-03-24T11:49:28.848565+0000","last_active":"2024-03-24T11:49:57.139963+0000","last_peered":"2024-03-24T11:49:57.139963+0000","last_clean":"2024-03-24T11:49:57.139963+0000","last_became_active":"2024-03-24T11:49:28.848025+0000","last_became_peered":"2024-03-24T11:49:28.848025+0000","last_unstale":"2024-03-24T11:49:57.139963+0000","last_undegraded":"2024-03-24T11:49:57.139963+0000","last_fullsized":"2024-03-24T11:49:57.139963+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:11:40.434654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140274+0000","last_change":"2024-03-24T11:49:32.881554+0000","last_active":"2024-03-24T11:49:57.140274+0000","last_peered":"2024-03-24T11:49:57.140274+0000","last_clean":"2024-03-24T11:49:57.140274+0000","last_became_active":"2024-03-24T11:49:32.880618+0000","last_became_peered":"2024-03-24T11:49:32.880618+0000","last_unstale":"2024-03-24T11:49:57.140274+0000","last_undegraded":"2024-03-24T11:49:57.140274+0000","last_fullsized":"2024-03-24T11:49:57.140274+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:16.260169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.142122+0000","last_change":"2024-03-24T11:49:30.869188+0000","last_active":"2024-03-24T11:49:57.142122+0000","last_peered":"2024-03-24T11:49:57.142122+0000","last_clean":"2024-03-24T11:49:57.142122+0000","last_became_active":"2024-03-24T11:49:30.868877+0000","last_became_peered":"2024-03-24T11:49:30.868877+0000","last_unstale":"2024-03-24T11:49:57.142122+0000","last_undegraded":"2024-03-24T11:49:57.142122+0000","last_fullsized":"2024-03-24T11:49:57.142122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:26:51.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138940+0000","last_change":"2024-03-24T11:49:26.831243+0000","last_active":"2024-03-24T11:49:57.138940+0000","last_peered":"2024-03-24T11:49:57.138940+0000","last_clean":"2024-03-24T11:49:57.138940+0000","last_became_active":"2024-03-24T11:49:26.830403+0000","last_became_peered":"2024-03-24T11:49:26.830403+0000","last_unstale":"2024-03-24T11:49:57.138940+0000","last_undegraded":"2024-03-24T11:49:57.138940+0000","last_fullsized":"2024-03-24T11:49:57.138940+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:04:29.428630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138857+0000","last_change":"2024-03-24T11:49:28.850593+0000","last_active":"2024-03-24T11:49:57.138857+0000","last_peered":"2024-03-24T11:49:57.138857+0000","last_clean":"2024-03-24T11:49:57.138857+0000","last_became_active":"2024-03-24T11:49:28.850298+0000","last_became_peered":"2024-03-24T11:49:28.850298+0000","last_unstale":"2024-03-24T11:49:57.138857+0000","last_undegraded":"2024-03-24T11:49:57.138857+0000","last_fullsized":"2024-03-24T11:49:57.138857+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:29.430524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141987+0000","last_change":"2024-03-24T11:49:32.884153+0000","last_active":"2024-03-24T11:49:57.141987+0000","last_peered":"2024-03-24T11:49:57.141987+0000","last_clean":"2024-03-24T11:49:57.141987+0000","last_became_active":"2024-03-24T11:49:32.883704+0000","last_became_peered":"2024-03-24T11:49:32.883704+0000","last_unstale":"2024-03-24T11:49:57.141987+0000","last_undegraded":"2024-03-24T11:49:57.141987+0000","last_fullsized":"2024-03-24T11:49:57.141987+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:15:17.917106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140433+0000","last_change":"2024-03-24T11:49:30.868090+0000","last_active":"2024-03-24T11:49:57.140433+0000","last_peered":"2024-03-24T11:49:57.140433+0000","last_clean":"2024-03-24T11:49:57.140433+0000","last_became_active":"2024-03-24T11:49:30.867329+0000","last_became_peered":"2024-03-24T11:49:30.867329+0000","last_unstale":"2024-03-24T11:49:57.140433+0000","last_undegraded":"2024-03-24T11:49:57.140433+0000","last_fullsized":"2024-03-24T11:49:57.140433+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:57:17.526006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139047+0000","last_change":"2024-03-24T11:49:26.837022+0000","last_active":"2024-03-24T11:49:57.139047+0000","last_peered":"2024-03-24T11:49:57.139047+0000","last_clean":"2024-03-24T11:49:57.139047+0000","last_became_active":"2024-03-24T11:49:26.834790+0000","last_became_peered":"2024-03-24T11:49:26.834790+0000","last_unstale":"2024-03-24T11:49:57.139047+0000","last_undegraded":"2024-03-24T11:49:57.139047+0000","last_fullsized":"2024-03-24T11:49:57.139047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:33.082967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139078+0000","last_change":"2024-03-24T11:49:28.850485+0000","last_active":"2024-03-24T11:49:57.139078+0000","last_peered":"2024-03-24T11:49:57.139078+0000","last_clean":"2024-03-24T11:49:57.139078+0000","last_became_active":"2024-03-24T11:49:28.849659+0000","last_became_peered":"2024-03-24T11:49:28.849659+0000","last_unstale":"2024-03-24T11:49:57.139078+0000","last_undegraded":"2024-03-24T11:49:57.139078+0000","last_fullsized":"2024-03-24T11:49:57.139078+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:12:47.288853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139629+0000","last_change":"2024-03-24T11:49:32.884915+0000","last_active":"2024-03-24T11:49:57.139629+0000","last_peered":"2024-03-24T11:49:57.139629+0000","last_clean":"2024-03-24T11:49:57.139629+0000","last_became_active":"2024-03-24T11:49:32.884604+0000","last_became_peered":"2024-03-24T11:49:32.884604+0000","last_unstale":"2024-03-24T11:49:57.139629+0000","last_undegraded":"2024-03-24T11:49:57.139629+0000","last_fullsized":"2024-03-24T11:49:57.139629+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:43.313298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140506+0000","last_change":"2024-03-24T11:49:26.834688+0000","last_active":"2024-03-24T11:49:57.140506+0000","last_peered":"2024-03-24T11:49:57.140506+0000","last_clean":"2024-03-24T11:49:57.140506+0000","last_became_active":"2024-03-24T11:49:26.833929+0000","last_became_peered":"2024-03-24T11:49:26.833929+0000","last_unstale":"2024-03-24T11:49:57.140506+0000","last_undegraded":"2024-03-24T11:49:57.140506+0000","last_fullsized":"2024-03-24T11:49:57.140506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:03:52.862106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140704+0000","last_change":"2024-03-24T11:49:28.850282+0000","last_active":"2024-03-24T11:49:57.140704+0000","last_peered":"2024-03-24T11:49:57.140704+0000","last_clean":"2024-03-24T11:49:57.140704+0000","last_became_active":"2024-03-24T11:49:28.849994+0000","last_became_peered":"2024-03-24T11:49:28.849994+0000","last_unstale":"2024-03-24T11:49:57.140704+0000","last_undegraded":"2024-03-24T11:49:57.140704+0000","last_fullsized":"2024-03-24T11:49:57.140704+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:07:20.366273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:50:01.879649+0000","last_change":"2024-03-24T11:49:30.859104+0000","last_active":"2024-03-24T11:50:01.879649+0000","last_peered":"2024-03-24T11:50:01.879649+0000","last_clean":"2024-03-24T11:50:01.879649+0000","last_became_active":"2024-03-24T11:49:30.858779+0000","last_became_peered":"2024-03-24T11:49:30.858779+0000","last_unstale":"2024-03-24T11:50:01.879649+0000","last_undegraded":"2024-03-24T11:50:01.879649+0000","last_fullsized":"2024-03-24T11:50:01.879649+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:51:45.455282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139759+0000","last_change":"2024-03-24T11:49:32.881363+0000","last_active":"2024-03-24T11:49:57.139759+0000","last_peered":"2024-03-24T11:49:57.139759+0000","last_clean":"2024-03-24T11:49:57.139759+0000","last_became_active":"2024-03-24T11:49:32.881023+0000","last_became_peered":"2024-03-24T11:49:32.881023+0000","last_unstale":"2024-03-24T11:49:57.139759+0000","last_undegraded":"2024-03-24T11:49:57.139759+0000","last_fullsized":"2024-03-24T11:49:57.139759+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:54:43.612364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141361+0000","last_change":"2024-03-24T11:49:26.840208+0000","last_active":"2024-03-24T11:49:57.141361+0000","last_peered":"2024-03-24T11:49:57.141361+0000","last_clean":"2024-03-24T11:49:57.141361+0000","last_became_active":"2024-03-24T11:49:26.839960+0000","last_became_peered":"2024-03-24T11:49:26.839960+0000","last_unstale":"2024-03-24T11:49:57.141361+0000","last_undegraded":"2024-03-24T11:49:57.141361+0000","last_fullsized":"2024-03-24T11:49:57.141361+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:05:59.317091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:50:01.879637+0000","last_change":"2024-03-24T11:49:28.848929+0000","last_active":"2024-03-24T11:50:01.879637+0000","last_peered":"2024-03-24T11:50:01.879637+0000","last_clean":"2024-03-24T11:50:01.879637+0000","last_became_active":"2024-03-24T11:49:28.848494+0000","last_became_peered":"2024-03-24T11:49:28.848494+0000","last_unstale":"2024-03-24T11:50:01.879637+0000","last_undegraded":"2024-03-24T11:50:01.879637+0000","last_fullsized":"2024-03-24T11:50:01.879637+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:35:20.301258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138969+0000","last_change":"2024-03-24T11:49:30.860547+0000","last_active":"2024-03-24T11:49:57.138969+0000","last_peered":"2024-03-24T11:49:57.138969+0000","last_clean":"2024-03-24T11:49:57.138969+0000","last_became_active":"2024-03-24T11:49:30.859905+0000","last_became_peered":"2024-03-24T11:49:30.859905+0000","last_unstale":"2024-03-24T11:49:57.138969+0000","last_undegraded":"2024-03-24T11:49:57.138969+0000","last_fullsized":"2024-03-24T11:49:57.138969+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:29:46.368494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140290+0000","last_change":"2024-03-24T11:49:32.883820+0000","last_active":"2024-03-24T11:49:57.140290+0000","last_peered":"2024-03-24T11:49:57.140290+0000","last_clean":"2024-03-24T11:49:57.140290+0000","last_became_active":"2024-03-24T11:49:32.883334+0000","last_became_peered":"2024-03-24T11:49:32.883334+0000","last_unstale":"2024-03-24T11:49:57.140290+0000","last_undegraded":"2024-03-24T11:49:57.140290+0000","last_fullsized":"2024-03-24T11:49:57.140290+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:15.107756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139738+0000","last_change":"2024-03-24T11:49:26.831325+0000","last_active":"2024-03-24T11:49:57.139738+0000","last_peered":"2024-03-24T11:49:57.139738+0000","last_clean":"2024-03-24T11:49:57.139738+0000","last_became_active":"2024-03-24T11:49:26.830449+0000","last_became_peered":"2024-03-24T11:49:26.830449+0000","last_unstale":"2024-03-24T11:49:57.139738+0000","last_undegraded":"2024-03-24T11:49:57.139738+0000","last_fullsized":"2024-03-24T11:49:57.139738+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:42.502768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140401+0000","last_change":"2024-03-24T11:49:28.850435+0000","last_active":"2024-03-24T11:49:57.140401+0000","last_peered":"2024-03-24T11:49:57.140401+0000","last_clean":"2024-03-24T11:49:57.140401+0000","last_became_active":"2024-03-24T11:49:28.850162+0000","last_became_peered":"2024-03-24T11:49:28.850162+0000","last_unstale":"2024-03-24T11:49:57.140401+0000","last_undegraded":"2024-03-24T11:49:57.140401+0000","last_fullsized":"2024-03-24T11:49:57.140401+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:37:53.163817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140544+0000","last_change":"2024-03-24T11:49:30.867938+0000","last_active":"2024-03-24T11:49:57.140544+0000","last_peered":"2024-03-24T11:49:57.140544+0000","last_clean":"2024-03-24T11:49:57.140544+0000","last_became_active":"2024-03-24T11:49:30.867128+0000","last_became_peered":"2024-03-24T11:49:30.867128+0000","last_unstale":"2024-03-24T11:49:57.140544+0000","last_undegraded":"2024-03-24T11:49:57.140544+0000","last_fullsized":"2024-03-24T11:49:57.140544+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:48:47.967469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141896+0000","last_change":"2024-03-24T11:49:32.884428+0000","last_active":"2024-03-24T11:49:57.141896+0000","last_peered":"2024-03-24T11:49:57.141896+0000","last_clean":"2024-03-24T11:49:57.141896+0000","last_became_active":"2024-03-24T11:49:32.884000+0000","last_became_peered":"2024-03-24T11:49:32.884000+0000","last_unstale":"2024-03-24T11:49:57.141896+0000","last_undegraded":"2024-03-24T11:49:57.141896+0000","last_fullsized":"2024-03-24T11:49:57.141896+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:58:05.113766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'18","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138522+0000","last_change":"2024-03-24T11:49:28.851547+0000","last_active":"2024-03-24T11:49:57.138522+0000","last_peered":"2024-03-24T11:49:57.138522+0000","last_clean":"2024-03-24T11:49:57.138522+0000","last_became_active":"2024-03-24T11:49:28.851174+0000","last_became_peered":"2024-03-24T11:49:28.851174+0000","last_unstale":"2024-03-24T11:49:57.138522+0000","last_undegraded":"2024-03-24T11:49:57.138522+0000","last_fullsized":"2024-03-24T11:49:57.138522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:39:11.038726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140053+0000","last_change":"2024-03-24T11:49:26.835454+0000","last_active":"2024-03-24T11:49:57.140053+0000","last_peered":"2024-03-24T11:49:57.140053+0000","last_clean":"2024-03-24T11:49:57.140053+0000","last_became_active":"2024-03-24T11:49:26.835117+0000","last_became_peered":"2024-03-24T11:49:26.835117+0000","last_unstale":"2024-03-24T11:49:57.140053+0000","last_undegraded":"2024-03-24T11:49:57.140053+0000","last_fullsized":"2024-03-24T11:49:57.140053+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:09:50.187146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.143286+0000","last_change":"2024-03-24T11:49:30.869424+0000","last_active":"2024-03-24T11:49:57.143286+0000","last_peered":"2024-03-24T11:49:57.143286+0000","last_clean":"2024-03-24T11:49:57.143286+0000","last_became_active":"2024-03-24T11:49:30.869059+0000","last_became_peered":"2024-03-24T11:49:30.869059+0000","last_unstale":"2024-03-24T11:49:57.143286+0000","last_undegraded":"2024-03-24T11:49:57.143286+0000","last_fullsized":"2024-03-24T11:49:57.143286+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:03.244382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139672+0000","last_change":"2024-03-24T11:49:32.881811+0000","last_active":"2024-03-24T11:49:57.139672+0000","last_peered":"2024-03-24T11:49:57.139672+0000","last_clean":"2024-03-24T11:49:57.139672+0000","last_became_active":"2024-03-24T11:49:32.881493+0000","last_became_peered":"2024-03-24T11:49:32.881493+0000","last_unstale":"2024-03-24T11:49:57.139672+0000","last_undegraded":"2024-03-24T11:49:57.139672+0000","last_fullsized":"2024-03-24T11:49:57.139672+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:07:26.879042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141170+0000","last_change":"2024-03-24T11:49:28.851363+0000","last_active":"2024-03-24T11:49:57.141170+0000","last_peered":"2024-03-24T11:49:57.141170+0000","last_clean":"2024-03-24T11:49:57.141170+0000","last_became_active":"2024-03-24T11:49:28.850236+0000","last_became_peered":"2024-03-24T11:49:28.850236+0000","last_unstale":"2024-03-24T11:49:57.141170+0000","last_undegraded":"2024-03-24T11:49:57.141170+0000","last_fullsized":"2024-03-24T11:49:57.141170+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:21:17.101167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139974+0000","last_change":"2024-03-24T11:49:26.831102+0000","last_active":"2024-03-24T11:49:57.139974+0000","last_peered":"2024-03-24T11:49:57.139974+0000","last_clean":"2024-03-24T11:49:57.139974+0000","last_became_active":"2024-03-24T11:49:26.830182+0000","last_became_peered":"2024-03-24T11:49:26.830182+0000","last_unstale":"2024-03-24T11:49:57.139974+0000","last_undegraded":"2024-03-24T11:49:57.139974+0000","last_fullsized":"2024-03-24T11:49:57.139974+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:12:41.734925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:59.236671+0000","last_change":"2024-03-24T11:49:16.896490+0000","last_active":"2024-03-24T11:49:59.236671+0000","last_peered":"2024-03-24T11:49:59.236671+0000","last_clean":"2024-03-24T11:49:59.236671+0000","last_became_active":"2024-03-24T11:49:16.895549+0000","last_became_peered":"2024-03-24T11:49:16.895549+0000","last_unstale":"2024-03-24T11:49:59.236671+0000","last_undegraded":"2024-03-24T11:49:59.236671+0000","last_fullsized":"2024-03-24T11:49:59.236671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:15.880507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:15.880507+0000","last_clean_scrub_stamp":"2024-03-24T11:49:15.880507+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:02:42.558300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140837+0000","last_change":"2024-03-24T11:49:30.863746+0000","last_active":"2024-03-24T11:49:57.140837+0000","last_peered":"2024-03-24T11:49:57.140837+0000","last_clean":"2024-03-24T11:49:57.140837+0000","last_became_active":"2024-03-24T11:49:30.863357+0000","last_became_peered":"2024-03-24T11:49:30.863357+0000","last_unstale":"2024-03-24T11:49:57.140837+0000","last_undegraded":"2024-03-24T11:49:57.140837+0000","last_fullsized":"2024-03-24T11:49:57.140837+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:05:37.110571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140407+0000","last_change":"2024-03-24T11:49:32.883264+0000","last_active":"2024-03-24T11:49:57.140407+0000","last_peered":"2024-03-24T11:49:57.140407+0000","last_clean":"2024-03-24T11:49:57.140407+0000","last_became_active":"2024-03-24T11:49:32.882778+0000","last_became_peered":"2024-03-24T11:49:32.882778+0000","last_unstale":"2024-03-24T11:49:57.140407+0000","last_undegraded":"2024-03-24T11:49:57.140407+0000","last_fullsized":"2024-03-24T11:49:57.140407+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:44:06.868622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'14","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140481+0000","last_change":"2024-03-24T11:49:28.850342+0000","last_active":"2024-03-24T11:49:57.140481+0000","last_peered":"2024-03-24T11:49:57.140481+0000","last_clean":"2024-03-24T11:49:57.140481+0000","last_became_active":"2024-03-24T11:49:28.849889+0000","last_became_peered":"2024-03-24T11:49:28.849889+0000","last_unstale":"2024-03-24T11:49:57.140481+0000","last_undegraded":"2024-03-24T11:49:57.140481+0000","last_fullsized":"2024-03-24T11:49:57.140481+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:18:08.270045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138809+0000","last_change":"2024-03-24T11:49:26.835111+0000","last_active":"2024-03-24T11:49:57.138809+0000","last_peered":"2024-03-24T11:49:57.138809+0000","last_clean":"2024-03-24T11:49:57.138809+0000","last_became_active":"2024-03-24T11:49:26.834742+0000","last_became_peered":"2024-03-24T11:49:26.834742+0000","last_unstale":"2024-03-24T11:49:57.138809+0000","last_undegraded":"2024-03-24T11:49:57.138809+0000","last_fullsized":"2024-03-24T11:49:57.138809+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:58.677268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140518+0000","last_change":"2024-03-24T11:49:30.860150+0000","last_active":"2024-03-24T11:49:57.140518+0000","last_peered":"2024-03-24T11:49:57.140518+0000","last_clean":"2024-03-24T11:49:57.140518+0000","last_became_active":"2024-03-24T11:49:30.859839+0000","last_became_peered":"2024-03-24T11:49:30.859839+0000","last_unstale":"2024-03-24T11:49:57.140518+0000","last_undegraded":"2024-03-24T11:49:57.140518+0000","last_fullsized":"2024-03-24T11:49:57.140518+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:31:13.190445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139521+0000","last_change":"2024-03-24T11:49:32.881406+0000","last_active":"2024-03-24T11:49:57.139521+0000","last_peered":"2024-03-24T11:49:57.139521+0000","last_clean":"2024-03-24T11:49:57.139521+0000","last_became_active":"2024-03-24T11:49:32.881016+0000","last_became_peered":"2024-03-24T11:49:32.881016+0000","last_unstale":"2024-03-24T11:49:57.139521+0000","last_undegraded":"2024-03-24T11:49:57.139521+0000","last_fullsized":"2024-03-24T11:49:57.139521+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:10:49.300161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141370+0000","last_change":"2024-03-24T11:49:26.833033+0000","last_active":"2024-03-24T11:49:57.141370+0000","last_peered":"2024-03-24T11:49:57.141370+0000","last_clean":"2024-03-24T11:49:57.141370+0000","last_became_active":"2024-03-24T11:49:26.832733+0000","last_became_peered":"2024-03-24T11:49:26.832733+0000","last_unstale":"2024-03-24T11:49:57.141370+0000","last_undegraded":"2024-03-24T11:49:57.141370+0000","last_fullsized":"2024-03-24T11:49:57.141370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:58:42.173192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'28","reported_seq":80,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:50:01.879375+0000","last_change":"2024-03-24T11:49:28.851594+0000","last_active":"2024-03-24T11:50:01.879375+0000","last_peered":"2024-03-24T11:50:01.879375+0000","last_clean":"2024-03-24T11:50:01.879375+0000","last_became_active":"2024-03-24T11:49:28.851268+0000","last_became_peered":"2024-03-24T11:49:28.851268+0000","last_unstale":"2024-03-24T11:50:01.879375+0000","last_undegraded":"2024-03-24T11:50:01.879375+0000","last_fullsized":"2024-03-24T11:50:01.879375+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:50:23.249657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:50:01.879603+0000","last_change":"2024-03-24T11:49:30.860075+0000","last_active":"2024-03-24T11:50:01.879603+0000","last_peered":"2024-03-24T11:50:01.879603+0000","last_clean":"2024-03-24T11:50:01.879603+0000","last_became_active":"2024-03-24T11:49:30.859742+0000","last_became_peered":"2024-03-24T11:49:30.859742+0000","last_unstale":"2024-03-24T11:50:01.879603+0000","last_undegraded":"2024-03-24T11:50:01.879603+0000","last_fullsized":"2024-03-24T11:50:01.879603+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:30:42.314971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138552+0000","last_change":"2024-03-24T11:49:32.882459+0000","last_active":"2024-03-24T11:49:57.138552+0000","last_peered":"2024-03-24T11:49:57.138552+0000","last_clean":"2024-03-24T11:49:57.138552+0000","last_became_active":"2024-03-24T11:49:32.881768+0000","last_became_peered":"2024-03-24T11:49:32.881768+0000","last_unstale":"2024-03-24T11:49:57.138552+0000","last_undegraded":"2024-03-24T11:49:57.138552+0000","last_fullsized":"2024-03-24T11:49:57.138552+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:28:06.013133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141664+0000","last_change":"2024-03-24T11:49:26.836256+0000","last_active":"2024-03-24T11:49:57.141664+0000","last_peered":"2024-03-24T11:49:57.141664+0000","last_clean":"2024-03-24T11:49:57.141664+0000","last_became_active":"2024-03-24T11:49:26.835548+0000","last_became_peered":"2024-03-24T11:49:26.835548+0000","last_unstale":"2024-03-24T11:49:57.141664+0000","last_undegraded":"2024-03-24T11:49:57.141664+0000","last_fullsized":"2024-03-24T11:49:57.141664+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:07:22.041247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.143436+0000","last_change":"2024-03-24T11:49:28.853307+0000","last_active":"2024-03-24T11:49:57.143436+0000","last_peered":"2024-03-24T11:49:57.143436+0000","last_clean":"2024-03-24T11:49:57.143436+0000","last_became_active":"2024-03-24T11:49:28.852918+0000","last_became_peered":"2024-03-24T11:49:28.852918+0000","last_unstale":"2024-03-24T11:49:57.143436+0000","last_undegraded":"2024-03-24T11:49:57.143436+0000","last_fullsized":"2024-03-24T11:49:57.143436+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:52:19.170963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:50:01.879373+0000","last_change":"2024-03-24T11:49:30.864224+0000","last_active":"2024-03-24T11:50:01.879373+0000","last_peered":"2024-03-24T11:50:01.879373+0000","last_clean":"2024-03-24T11:50:01.879373+0000","last_became_active":"2024-03-24T11:49:30.863940+0000","last_became_peered":"2024-03-24T11:49:30.863940+0000","last_unstale":"2024-03-24T11:50:01.879373+0000","last_undegraded":"2024-03-24T11:50:01.879373+0000","last_fullsized":"2024-03-24T11:50:01.879373+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:32:17.040409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139533+0000","last_change":"2024-03-24T11:49:32.880662+0000","last_active":"2024-03-24T11:49:57.139533+0000","last_peered":"2024-03-24T11:49:57.139533+0000","last_clean":"2024-03-24T11:49:57.139533+0000","last_became_active":"2024-03-24T11:49:32.880384+0000","last_became_peered":"2024-03-24T11:49:32.880384+0000","last_unstale":"2024-03-24T11:49:57.139533+0000","last_undegraded":"2024-03-24T11:49:57.139533+0000","last_fullsized":"2024-03-24T11:49:57.139533+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:42:13.999971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141081+0000","last_change":"2024-03-24T11:49:26.834924+0000","last_active":"2024-03-24T11:49:57.141081+0000","last_peered":"2024-03-24T11:49:57.141081+0000","last_clean":"2024-03-24T11:49:57.141081+0000","last_became_active":"2024-03-24T11:49:26.834589+0000","last_became_peered":"2024-03-24T11:49:26.834589+0000","last_unstale":"2024-03-24T11:49:57.141081+0000","last_undegraded":"2024-03-24T11:49:57.141081+0000","last_fullsized":"2024-03-24T11:49:57.141081+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:24.738910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141472+0000","last_change":"2024-03-24T11:49:28.850377+0000","last_active":"2024-03-24T11:49:57.141472+0000","last_peered":"2024-03-24T11:49:57.141472+0000","last_clean":"2024-03-24T11:49:57.141472+0000","last_became_active":"2024-03-24T11:49:28.849991+0000","last_became_peered":"2024-03-24T11:49:28.849991+0000","last_unstale":"2024-03-24T11:49:57.141472+0000","last_undegraded":"2024-03-24T11:49:57.141472+0000","last_fullsized":"2024-03-24T11:49:57.141472+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:38:46.705905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138654+0000","last_change":"2024-03-24T11:49:30.859125+0000","last_active":"2024-03-24T11:49:57.138654+0000","last_peered":"2024-03-24T11:49:57.138654+0000","last_clean":"2024-03-24T11:49:57.138654+0000","last_became_active":"2024-03-24T11:49:30.858867+0000","last_became_peered":"2024-03-24T11:49:30.858867+0000","last_unstale":"2024-03-24T11:49:57.138654+0000","last_undegraded":"2024-03-24T11:49:57.138654+0000","last_fullsized":"2024-03-24T11:49:57.138654+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:23:29.970739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140585+0000","last_change":"2024-03-24T11:49:32.882073+0000","last_active":"2024-03-24T11:49:57.140585+0000","last_peered":"2024-03-24T11:49:57.140585+0000","last_clean":"2024-03-24T11:49:57.140585+0000","last_became_active":"2024-03-24T11:49:32.881265+0000","last_became_peered":"2024-03-24T11:49:32.881265+0000","last_unstale":"2024-03-24T11:49:57.140585+0000","last_undegraded":"2024-03-24T11:49:57.140585+0000","last_fullsized":"2024-03-24T11:49:57.140585+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:01:23.527011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:50:01.879556+0000","last_change":"2024-03-24T11:49:30.866180+0000","last_active":"2024-03-24T11:50:01.879556+0000","last_peered":"2024-03-24T11:50:01.879556+0000","last_clean":"2024-03-24T11:50:01.879556+0000","last_became_active":"2024-03-24T11:49:30.865820+0000","last_became_peered":"2024-03-24T11:49:30.865820+0000","last_unstale":"2024-03-24T11:50:01.879556+0000","last_undegraded":"2024-03-24T11:50:01.879556+0000","last_fullsized":"2024-03-24T11:50:01.879556+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:07.099753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138753+0000","last_change":"2024-03-24T11:49:26.835688+0000","last_active":"2024-03-24T11:49:57.138753+0000","last_peered":"2024-03-24T11:49:57.138753+0000","last_clean":"2024-03-24T11:49:57.138753+0000","last_became_active":"2024-03-24T11:49:26.835244+0000","last_became_peered":"2024-03-24T11:49:26.835244+0000","last_unstale":"2024-03-24T11:49:57.138753+0000","last_undegraded":"2024-03-24T11:49:57.138753+0000","last_fullsized":"2024-03-24T11:49:57.138753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:39:39.385060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139216+0000","last_change":"2024-03-24T11:49:28.850122+0000","last_active":"2024-03-24T11:49:57.139216+0000","last_peered":"2024-03-24T11:49:57.139216+0000","last_clean":"2024-03-24T11:49:57.139216+0000","last_became_active":"2024-03-24T11:49:28.849817+0000","last_became_peered":"2024-03-24T11:49:28.849817+0000","last_unstale":"2024-03-24T11:49:57.139216+0000","last_undegraded":"2024-03-24T11:49:57.139216+0000","last_fullsized":"2024-03-24T11:49:57.139216+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:39:40.308805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141333+0000","last_change":"2024-03-24T11:49:32.882010+0000","last_active":"2024-03-24T11:49:57.141333+0000","last_peered":"2024-03-24T11:49:57.141333+0000","last_clean":"2024-03-24T11:49:57.141333+0000","last_became_active":"2024-03-24T11:49:32.881696+0000","last_became_peered":"2024-03-24T11:49:32.881696+0000","last_unstale":"2024-03-24T11:49:57.141333+0000","last_undegraded":"2024-03-24T11:49:57.141333+0000","last_fullsized":"2024-03-24T11:49:57.141333+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:58:35.897234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.142147+0000","last_change":"2024-03-24T11:49:30.868328+0000","last_active":"2024-03-24T11:49:57.142147+0000","last_peered":"2024-03-24T11:49:57.142147+0000","last_clean":"2024-03-24T11:49:57.142147+0000","last_became_active":"2024-03-24T11:49:30.867467+0000","last_became_peered":"2024-03-24T11:49:30.867467+0000","last_unstale":"2024-03-24T11:49:57.142147+0000","last_undegraded":"2024-03-24T11:49:57.142147+0000","last_fullsized":"2024-03-24T11:49:57.142147+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:18:46.136182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139873+0000","last_change":"2024-03-24T11:49:26.836326+0000","last_active":"2024-03-24T11:49:57.139873+0000","last_peered":"2024-03-24T11:49:57.139873+0000","last_clean":"2024-03-24T11:49:57.139873+0000","last_became_active":"2024-03-24T11:49:26.836005+0000","last_became_peered":"2024-03-24T11:49:26.836005+0000","last_unstale":"2024-03-24T11:49:57.139873+0000","last_undegraded":"2024-03-24T11:49:57.139873+0000","last_fullsized":"2024-03-24T11:49:57.139873+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:49:41.808088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139161+0000","last_change":"2024-03-24T11:49:28.849954+0000","last_active":"2024-03-24T11:49:57.139161+0000","last_peered":"2024-03-24T11:49:57.139161+0000","last_clean":"2024-03-24T11:49:57.139161+0000","last_became_active":"2024-03-24T11:49:28.849663+0000","last_became_peered":"2024-03-24T11:49:28.849663+0000","last_unstale":"2024-03-24T11:49:57.139161+0000","last_undegraded":"2024-03-24T11:49:57.139161+0000","last_fullsized":"2024-03-24T11:49:57.139161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:49:58.706757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139957+0000","last_change":"2024-03-24T11:49:32.882079+0000","last_active":"2024-03-24T11:49:57.139957+0000","last_peered":"2024-03-24T11:49:57.139957+0000","last_clean":"2024-03-24T11:49:57.139957+0000","last_became_active":"2024-03-24T11:49:32.881805+0000","last_became_peered":"2024-03-24T11:49:32.881805+0000","last_unstale":"2024-03-24T11:49:57.139957+0000","last_undegraded":"2024-03-24T11:49:57.139957+0000","last_fullsized":"2024-03-24T11:49:57.139957+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:26:41.311574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139473+0000","last_change":"2024-03-24T11:49:26.837054+0000","last_active":"2024-03-24T11:49:57.139473+0000","last_peered":"2024-03-24T11:49:57.139473+0000","last_clean":"2024-03-24T11:49:57.139473+0000","last_became_active":"2024-03-24T11:49:26.836652+0000","last_became_peered":"2024-03-24T11:49:26.836652+0000","last_unstale":"2024-03-24T11:49:57.139473+0000","last_undegraded":"2024-03-24T11:49:57.139473+0000","last_fullsized":"2024-03-24T11:49:57.139473+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:41:18.299556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140346+0000","last_change":"2024-03-24T11:49:28.850275+0000","last_active":"2024-03-24T11:49:57.140346+0000","last_peered":"2024-03-24T11:49:57.140346+0000","last_clean":"2024-03-24T11:49:57.140346+0000","last_became_active":"2024-03-24T11:49:28.849729+0000","last_became_peered":"2024-03-24T11:49:28.849729+0000","last_unstale":"2024-03-24T11:49:57.140346+0000","last_undegraded":"2024-03-24T11:49:57.140346+0000","last_fullsized":"2024-03-24T11:49:57.140346+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:22:50.537818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140245+0000","last_change":"2024-03-24T11:49:30.860562+0000","last_active":"2024-03-24T11:49:57.140245+0000","last_peered":"2024-03-24T11:49:57.140245+0000","last_clean":"2024-03-24T11:49:57.140245+0000","last_became_active":"2024-03-24T11:49:30.859689+0000","last_became_peered":"2024-03-24T11:49:30.859689+0000","last_unstale":"2024-03-24T11:49:57.140245+0000","last_undegraded":"2024-03-24T11:49:57.140245+0000","last_fullsized":"2024-03-24T11:49:57.140245+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:34:11.801292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139577+0000","last_change":"2024-03-24T11:49:32.883684+0000","last_active":"2024-03-24T11:49:57.139577+0000","last_peered":"2024-03-24T11:49:57.139577+0000","last_clean":"2024-03-24T11:49:57.139577+0000","last_became_active":"2024-03-24T11:49:32.883301+0000","last_became_peered":"2024-03-24T11:49:32.883301+0000","last_unstale":"2024-03-24T11:49:57.139577+0000","last_undegraded":"2024-03-24T11:49:57.139577+0000","last_fullsized":"2024-03-24T11:49:57.139577+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:10:43.995441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:50:01.879630+0000","last_change":"2024-03-24T11:49:30.865157+0000","last_active":"2024-03-24T11:50:01.879630+0000","last_peered":"2024-03-24T11:50:01.879630+0000","last_clean":"2024-03-24T11:50:01.879630+0000","last_became_active":"2024-03-24T11:49:30.864685+0000","last_became_peered":"2024-03-24T11:49:30.864685+0000","last_unstale":"2024-03-24T11:50:01.879630+0000","last_undegraded":"2024-03-24T11:50:01.879630+0000","last_fullsized":"2024-03-24T11:50:01.879630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:52:04.753824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139764+0000","last_change":"2024-03-24T11:49:26.832963+0000","last_active":"2024-03-24T11:49:57.139764+0000","last_peered":"2024-03-24T11:49:57.139764+0000","last_clean":"2024-03-24T11:49:57.139764+0000","last_became_active":"2024-03-24T11:49:26.832354+0000","last_became_peered":"2024-03-24T11:49:26.832354+0000","last_unstale":"2024-03-24T11:49:57.139764+0000","last_undegraded":"2024-03-24T11:49:57.139764+0000","last_fullsized":"2024-03-24T11:49:57.139764+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:56:17.366171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'19","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139368+0000","last_change":"2024-03-24T11:49:28.848678+0000","last_active":"2024-03-24T11:49:57.139368+0000","last_peered":"2024-03-24T11:49:57.139368+0000","last_clean":"2024-03-24T11:49:57.139368+0000","last_became_active":"2024-03-24T11:49:28.847873+0000","last_became_peered":"2024-03-24T11:49:28.847873+0000","last_unstale":"2024-03-24T11:49:57.139368+0000","last_undegraded":"2024-03-24T11:49:57.139368+0000","last_fullsized":"2024-03-24T11:49:57.139368+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:17:48.243747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139105+0000","last_change":"2024-03-24T11:49:32.880703+0000","last_active":"2024-03-24T11:49:57.139105+0000","last_peered":"2024-03-24T11:49:57.139105+0000","last_clean":"2024-03-24T11:49:57.139105+0000","last_became_active":"2024-03-24T11:49:32.880435+0000","last_became_peered":"2024-03-24T11:49:32.880435+0000","last_unstale":"2024-03-24T11:49:57.139105+0000","last_undegraded":"2024-03-24T11:49:57.139105+0000","last_fullsized":"2024-03-24T11:49:57.139105+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:45:08.074039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140411+0000","last_change":"2024-03-24T11:49:26.831463+0000","last_active":"2024-03-24T11:49:57.140411+0000","last_peered":"2024-03-24T11:49:57.140411+0000","last_clean":"2024-03-24T11:49:57.140411+0000","last_became_active":"2024-03-24T11:49:26.831106+0000","last_became_peered":"2024-03-24T11:49:26.831106+0000","last_unstale":"2024-03-24T11:49:57.140411+0000","last_undegraded":"2024-03-24T11:49:57.140411+0000","last_fullsized":"2024-03-24T11:49:57.140411+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:04.719567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141149+0000","last_change":"2024-03-24T11:49:28.850007+0000","last_active":"2024-03-24T11:49:57.141149+0000","last_peered":"2024-03-24T11:49:57.141149+0000","last_clean":"2024-03-24T11:49:57.141149+0000","last_became_active":"2024-03-24T11:49:28.848956+0000","last_became_peered":"2024-03-24T11:49:28.848956+0000","last_unstale":"2024-03-24T11:49:57.141149+0000","last_undegraded":"2024-03-24T11:49:57.141149+0000","last_fullsized":"2024-03-24T11:49:57.141149+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:32.532840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139195+0000","last_change":"2024-03-24T11:49:30.860370+0000","last_active":"2024-03-24T11:49:57.139195+0000","last_peered":"2024-03-24T11:49:57.139195+0000","last_clean":"2024-03-24T11:49:57.139195+0000","last_became_active":"2024-03-24T11:49:30.859522+0000","last_became_peered":"2024-03-24T11:49:30.859522+0000","last_unstale":"2024-03-24T11:49:57.139195+0000","last_undegraded":"2024-03-24T11:49:57.139195+0000","last_fullsized":"2024-03-24T11:49:57.139195+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:48:35.511349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139380+0000","last_change":"2024-03-24T11:49:32.881195+0000","last_active":"2024-03-24T11:49:57.139380+0000","last_peered":"2024-03-24T11:49:57.139380+0000","last_clean":"2024-03-24T11:49:57.139380+0000","last_became_active":"2024-03-24T11:49:32.880785+0000","last_became_peered":"2024-03-24T11:49:32.880785+0000","last_unstale":"2024-03-24T11:49:57.139380+0000","last_undegraded":"2024-03-24T11:49:57.139380+0000","last_fullsized":"2024-03-24T11:49:57.139380+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:23.321147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141195+0000","last_change":"2024-03-24T11:49:26.836789+0000","last_active":"2024-03-24T11:49:57.141195+0000","last_peered":"2024-03-24T11:49:57.141195+0000","last_clean":"2024-03-24T11:49:57.141195+0000","last_became_active":"2024-03-24T11:49:26.835223+0000","last_became_peered":"2024-03-24T11:49:26.835223+0000","last_unstale":"2024-03-24T11:49:57.141195+0000","last_undegraded":"2024-03-24T11:49:57.141195+0000","last_fullsized":"2024-03-24T11:49:57.141195+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:04:45.325722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.140734+0000","last_change":"2024-03-24T11:49:28.850665+0000","last_active":"2024-03-24T11:49:57.140734+0000","last_peered":"2024-03-24T11:49:57.140734+0000","last_clean":"2024-03-24T11:49:57.140734+0000","last_became_active":"2024-03-24T11:49:28.850403+0000","last_became_peered":"2024-03-24T11:49:28.850403+0000","last_unstale":"2024-03-24T11:49:57.140734+0000","last_undegraded":"2024-03-24T11:49:57.140734+0000","last_fullsized":"2024-03-24T11:49:57.140734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:47.518866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139644+0000","last_change":"2024-03-24T11:49:30.867113+0000","last_active":"2024-03-24T11:49:57.139644+0000","last_peered":"2024-03-24T11:49:57.139644+0000","last_clean":"2024-03-24T11:49:57.139644+0000","last_became_active":"2024-03-24T11:49:30.866777+0000","last_became_peered":"2024-03-24T11:49:30.866777+0000","last_unstale":"2024-03-24T11:49:57.139644+0000","last_undegraded":"2024-03-24T11:49:57.139644+0000","last_fullsized":"2024-03-24T11:49:57.139644+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:48:35.183573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.138679+0000","last_change":"2024-03-24T11:49:32.881734+0000","last_active":"2024-03-24T11:49:57.138679+0000","last_peered":"2024-03-24T11:49:57.138679+0000","last_clean":"2024-03-24T11:49:57.138679+0000","last_became_active":"2024-03-24T11:49:32.880387+0000","last_became_peered":"2024-03-24T11:49:32.880387+0000","last_unstale":"2024-03-24T11:49:57.138679+0000","last_undegraded":"2024-03-24T11:49:57.138679+0000","last_fullsized":"2024-03-24T11:49:57.138679+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:31.861574+0000","last_clean_scrub_stamp":"2024-03-24T11:49:31.861574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:06.732552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141995+0000","last_change":"2024-03-24T11:49:26.836399+0000","last_active":"2024-03-24T11:49:57.141995+0000","last_peered":"2024-03-24T11:49:57.141995+0000","last_clean":"2024-03-24T11:49:57.141995+0000","last_became_active":"2024-03-24T11:49:26.836121+0000","last_became_peered":"2024-03-24T11:49:26.836121+0000","last_unstale":"2024-03-24T11:49:57.141995+0000","last_undegraded":"2024-03-24T11:49:57.141995+0000","last_fullsized":"2024-03-24T11:49:57.141995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:25.814730+0000","last_clean_scrub_stamp":"2024-03-24T11:49:25.814730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:01:16.583166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.141203+0000","last_change":"2024-03-24T11:49:28.849143+0000","last_active":"2024-03-24T11:49:57.141203+0000","last_peered":"2024-03-24T11:49:57.141203+0000","last_clean":"2024-03-24T11:49:57.141203+0000","last_became_active":"2024-03-24T11:49:28.848715+0000","last_became_peered":"2024-03-24T11:49:28.848715+0000","last_unstale":"2024-03-24T11:49:57.141203+0000","last_undegraded":"2024-03-24T11:49:57.141203+0000","last_fullsized":"2024-03-24T11:49:57.141203+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:27.831302+0000","last_clean_scrub_stamp":"2024-03-24T11:49:27.831302+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:26:44.048534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T11:49:57.139372+0000","last_change":"2024-03-24T11:49:30.865169+0000","last_active":"2024-03-24T11:49:57.139372+0000","last_peered":"2024-03-24T11:49:57.139372+0000","last_clean":"2024-03-24T11:49:57.139372+0000","last_became_active":"2024-03-24T11:49:30.864773+0000","last_became_peered":"2024-03-24T11:49:30.864773+0000","last_unstale":"2024-03-24T11:49:57.139372+0000","last_undegraded":"2024-03-24T11:49:57.139372+0000","last_fullsized":"2024-03-24T11:49:57.139372+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:49:29.843729+0000","last_clean_scrub_stamp":"2024-03-24T11:49:29.843729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:28:10.860288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708840,"statfs":{"total":95990841344,"available":95957852160,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738391,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708840,"statfs":{"total":95990841344,"available":95957852160,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708840,"statfs":{"total":95990841344,"available":95957852160,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:50:13.264 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T11:50:13.264 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T11:50:13.264 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T11:50:13.264 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T11:50:13.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:12 smithi174 ceph-mon[30177]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:50:13.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:12 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:13.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:12 smithi189 ceph-mon[30629]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:50:13.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:12 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:13.740 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:14.106 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:13 smithi130 ceph-mon[25795]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:50:14.344 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:13 smithi174 ceph-mon[30177]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:50:14.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:13 smithi189 ceph-mon[30629]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:50:15.189 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:14 smithi130 ceph-mon[25795]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:50:15.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:14 smithi174 ceph-mon[30177]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:50:15.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:14 smithi189 ceph-mon[30629]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:50:16.992 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:16 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: ::ffff:172.21.15.174 - - [24/Mar/2024:11:50:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:50:16.992 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:16 smithi130 podman[54466]: 2024-03-24 11:50:16.920818047 +0000 UTC m=+0.312397543 volume create 545f5dedeecc4b079c1b054f2c0b89a1707c24a74e691dde38a17331d35186a0 2024-03-24T11:50:17.274 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:16 smithi130 ceph-mon[25795]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:50:17.274 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 podman[54466]: 2024-03-24 11:50:17.004430937 +0000 UTC m=+0.396010391 container create 363631a85202c110ce6fd6d4eb889331900e265ff4dd5e1270f645b903ae6052 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:50:17.343 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:16 smithi174 ceph-mon[30177]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:50:17.445 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:16 smithi189 ceph-mon[30629]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:50:17.531 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 podman[54466]: 2024-03-24 11:50:17.483344845 +0000 UTC m=+0.874924300 container init 363631a85202c110ce6fd6d4eb889331900e265ff4dd5e1270f645b903ae6052 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:50:17.531 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 podman[54466]: 2024-03-24 11:50:17.488452681 +0000 UTC m=+0.880032135 container start 363631a85202c110ce6fd6d4eb889331900e265ff4dd5e1270f645b903ae6052 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:50:17.531 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a[54503]: ts=2024-03-24T11:50:17.499Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-24T11:50:17.531 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a[54503]: ts=2024-03-24T11:50:17.499Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-24T11:50:17.531 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a[54503]: ts=2024-03-24T11:50:17.501Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.130 port=9094 2024-03-24T11:50:17.532 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a[54503]: ts=2024-03-24T11:50:17.502Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T11:50:17.532 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a[54503]: ts=2024-03-24T11:50:17.531Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:50:17.532 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a[54503]: ts=2024-03-24T11:50:17.531Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:50:17.652 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-24T11:50:17.652 INFO:teuthology.orchestra.run.smithi130.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T11:50:17.788 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a[54503]: ts=2024-03-24T11:50:17.534Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-24T11:50:17.788 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a[54503]: ts=2024-03-24T11:50:17.534Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-24T11:50:17.788 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 bash[54466]: 363631a85202c110ce6fd6d4eb889331900e265ff4dd5e1270f645b903ae6052 2024-03-24T11:50:17.788 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:17 smithi130 systemd[1]: Started Ceph alertmanager.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:50:18.287 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:17] ENGINE Bus STOPPING 2024-03-24T11:50:18.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:50:18.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:17] ENGINE Bus STOPPED 2024-03-24T11:50:18.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:17] ENGINE Bus STARTING 2024-03-24T11:50:18.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:17] ENGINE Serving on http://:::9283 2024-03-24T11:50:18.288 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:17 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:17] ENGINE Bus STARTED 2024-03-24T11:50:18.323 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T11:50:18.323 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T11:50:18.323 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:50:18.334 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi130.front.sepia.ceph.com 2024-03-24T11:50:18.335 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-24T11:50:18.573 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:18.873 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:18.873 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: from='client.? 172.21.15.130:0/1312645769' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:50:18.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:50:18.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:50:18.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:50:18.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.874 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:18 smithi130 ceph-mon[25795]: Deploying daemon grafana.a on smithi174 2024-03-24T11:50:18.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:18.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: from='client.? 172.21.15.130:0/1312645769' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:50:18.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:50:18.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:50:18.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:50:18.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:18.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:18 smithi189 ceph-mon[30629]: Deploying daemon grafana.a on smithi174 2024-03-24T11:50:19.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: from='client.? 172.21.15.130:0/1312645769' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:19.094 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:18 smithi174 ceph-mon[30177]: Deploying daemon grafana.a on smithi174 2024-03-24T11:50:19.711 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:19 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a[54503]: ts=2024-03-24T11:50:19.503Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000839484s 2024-03-24T11:50:20.222 INFO:teuthology.orchestra.run.smithi130.stdout:167 167 2024-03-24T11:50:20.733 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:20 smithi130 ceph-mon[25795]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:20.734 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T11:50:20.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:20 smithi189 ceph-mon[30629]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:20.981 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:21.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:20 smithi174 ceph-mon[30177]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:22.749 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:22 smithi130 ceph-mon[25795]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:22.749 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:22 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:22.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:22 smithi189 ceph-mon[30629]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:22.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:22 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:23.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:22 smithi174 ceph-mon[30177]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:23.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:22 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:23.214 INFO:teuthology.orchestra.run.smithi130.stdout:Backend: cephadm 2024-03-24T11:50:23.215 INFO:teuthology.orchestra.run.smithi130.stdout:Available: Yes 2024-03-24T11:50:23.215 INFO:teuthology.orchestra.run.smithi130.stdout:Paused: No 2024-03-24T11:50:23.750 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T11:50:23.986 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:24.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:24 smithi189 ceph-mon[30629]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:24.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:24 smithi189 ceph-mon[30629]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:24.988 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:24 smithi130 ceph-mon[25795]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:24.988 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:24 smithi130 ceph-mon[25795]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:25.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:24 smithi174 ceph-mon[30177]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:25.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:24 smithi174 ceph-mon[30177]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:25.748 INFO:teuthology.orchestra.run.smithi130.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:50:25.748 INFO:teuthology.orchestra.run.smithi130.stdout:alertmanager.a smithi130 *:9093,9094 starting - - - - 2024-03-24T11:50:25.748 INFO:teuthology.orchestra.run.smithi130.stdout:mgr.x smithi174 *:8443,8765 running (2m) 26s ago 2m 447M - 19.0.0-1667-gdb0330b1 4f325f8440e9 3909890b4ca3 2024-03-24T11:50:25.748 INFO:teuthology.orchestra.run.smithi130.stdout:mgr.y smithi130 *:9283,8765 running (4m) 21s ago 4m 475M - 19.0.0-1667-gdb0330b1 4f325f8440e9 44df44b98973 2024-03-24T11:50:25.749 INFO:teuthology.orchestra.run.smithi130.stdout:mgr.z smithi189 *:8443,8765 running (2m) 26s ago 2m 447M - 19.0.0-1667-gdb0330b1 4f325f8440e9 78ee45e0c014 2024-03-24T11:50:25.749 INFO:teuthology.orchestra.run.smithi130.stdout:mon.a smithi130 running (4m) 21s ago 4m 47.3M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 360e2bed32ee 2024-03-24T11:50:25.749 INFO:teuthology.orchestra.run.smithi130.stdout:mon.b smithi174 running (2m) 26s ago 2m 42.0M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 fa9693ddc64f 2024-03-24T11:50:25.749 INFO:teuthology.orchestra.run.smithi130.stdout:mon.c smithi189 running (2m) 26s ago 2m 43.2M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 9df70d4d8c04 2024-03-24T11:50:25.749 INFO:teuthology.orchestra.run.smithi130.stdout:osd.0 smithi130 running (2m) 21s ago 2m 50.3M 15.6G 19.0.0-1667-gdb0330b1 4f325f8440e9 77c04ee66465 2024-03-24T11:50:25.749 INFO:teuthology.orchestra.run.smithi130.stdout:osd.1 smithi174 running (103s) 26s ago 102s 51.3M 15.6G 19.0.0-1667-gdb0330b1 4f325f8440e9 98ba3d93580c 2024-03-24T11:50:25.749 INFO:teuthology.orchestra.run.smithi130.stdout:osd.2 smithi189 running (78s) 26s ago 77s 49.9M 16.6G 19.0.0-1667-gdb0330b1 4f325f8440e9 dd47e0f1ecaf 2024-03-24T11:50:25.749 INFO:teuthology.orchestra.run.smithi130.stdout:prometheus.a smithi174 *:9095 running (37s) 26s ago 36s 27.9M - 2.43.0 a07b618ecd1d 6e5961b3423f 2024-03-24T11:50:25.749 INFO:teuthology.orchestra.run.smithi130.stdout:rgw.foo.a smithi130 *:80 running (60s) 21s ago 60s 78.5M - 19.0.0-1667-gdb0330b1 4f325f8440e9 b58dd8716230 2024-03-24T11:50:26.373 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T11:50:26.611 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:26.907 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:26 smithi130 ceph-mon[25795]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:26.907 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:26 smithi130 ceph-mon[25795]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:26.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:26 smithi189 ceph-mon[30629]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:26.946 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:26 smithi189 ceph-mon[30629]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:27.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:26 smithi174 ceph-mon[30177]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:27.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:26 smithi174 ceph-mon[30177]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:27.287 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:26 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: ::ffff:172.21.15.174 - - [24/Mar/2024:11:50:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:50:27.677 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:27 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:27.677 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:50:27 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a[54503]: ts=2024-03-24T11:50:27.505Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002857062s 2024-03-24T11:50:27.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:27 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:28.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:27 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:50:28.437 INFO:teuthology.orchestra.run.smithi130.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T11:50:28.437 INFO:teuthology.orchestra.run.smithi130.stdout:alertmanager ?:9093,9094 0/1 - 57s smithi130=a;count:1 2024-03-24T11:50:28.437 INFO:teuthology.orchestra.run.smithi130.stdout:grafana ?:3000 0/1 - 55s smithi174=a;count:1 2024-03-24T11:50:28.437 INFO:teuthology.orchestra.run.smithi130.stdout:mgr 3/3 29s ago 2m smithi130=y;smithi174=x;smithi189=z;count:3 2024-03-24T11:50:28.438 INFO:teuthology.orchestra.run.smithi130.stdout:mon 3/3 29s ago 3m smithi130:172.21.15.130=a;smithi174:172.21.15.174=b;smithi189:172.21.15.189=c;count:3 2024-03-24T11:50:28.438 INFO:teuthology.orchestra.run.smithi130.stdout:node-exporter ?:9100 0/3 - 60s smithi130=a;smithi174=b;smithi189=c;count:3 2024-03-24T11:50:28.438 INFO:teuthology.orchestra.run.smithi130.stdout:osd 3 29s ago - 2024-03-24T11:50:28.438 INFO:teuthology.orchestra.run.smithi130.stdout:prometheus ?:9095 1/1 28s ago 63s smithi174=a;count:1 2024-03-24T11:50:28.438 INFO:teuthology.orchestra.run.smithi130.stdout:rgw.foo.a ?:80 1/1 24s ago 62s smithi130=foo.a;count:1 2024-03-24T11:50:28.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:28 smithi189 ceph-mon[30629]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:28.992 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:28 smithi130 ceph-mon[25795]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:28.994 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:28 smithi174 ceph-mon[30177]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:29.019 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T11:50:29.256 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:29.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:29 smithi189 ceph-mon[30629]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:30.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:29 smithi174 ceph-mon[30177]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:30.016 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:29 smithi130 ceph-mon[25795]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:30.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:30 smithi189 ceph-mon[30629]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:31.037 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:30 smithi130 ceph-mon[25795]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:31.083 INFO:teuthology.orchestra.run.smithi130.stdout:HOST ADDR LABELS STATUS 2024-03-24T11:50:31.083 INFO:teuthology.orchestra.run.smithi130.stdout:smithi130 172.21.15.130 2024-03-24T11:50:31.083 INFO:teuthology.orchestra.run.smithi130.stdout:smithi174 172.21.15.174 2024-03-24T11:50:31.083 INFO:teuthology.orchestra.run.smithi130.stdout:smithi189 172.21.15.189 2024-03-24T11:50:31.083 INFO:teuthology.orchestra.run.smithi130.stdout:3 hosts in cluster 2024-03-24T11:50:31.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:30 smithi174 ceph-mon[30177]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:31.734 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T11:50:31.969 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:32.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:32 smithi189 ceph-mon[30629]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:32.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:32 smithi189 ceph-mon[30629]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:33.037 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:32 smithi130 ceph-mon[25795]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:33.037 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:32 smithi130 ceph-mon[25795]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:33.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:32 smithi174 ceph-mon[30177]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:33.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:32 smithi174 ceph-mon[30177]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:33.878 INFO:teuthology.orchestra.run.smithi130.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T11:50:33.878 INFO:teuthology.orchestra.run.smithi130.stdout:smithi130 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-24T11:50:33.878 INFO:teuthology.orchestra.run.smithi130.stdout:smithi130 /dev/nvme1n1 ssd Linux_2fb49f481a09d25465fc 89.4G Yes 2m ago 2024-03-24T11:50:33.878 INFO:teuthology.orchestra.run.smithi130.stdout:smithi130 /dev/nvme2n1 ssd Linux_ba58472c4149d39e9be8 89.4G Yes 2m ago 2024-03-24T11:50:33.878 INFO:teuthology.orchestra.run.smithi130.stdout:smithi130 /dev/nvme3n1 ssd Linux_56a859b84f523c3bf4b3 89.4G Yes 2m ago 2024-03-24T11:50:33.878 INFO:teuthology.orchestra.run.smithi130.stdout:smithi130 /dev/nvme4n1 ssd Linux_9194ecab66d47b0f253f 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-24T11:50:33.878 INFO:teuthology.orchestra.run.smithi130.stdout:smithi174 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050074400BGN 372G No 103s ago Has a FileSystem, LVM detected 2024-03-24T11:50:33.878 INFO:teuthology.orchestra.run.smithi130.stdout:smithi174 /dev/nvme1n1 ssd Linux_e085c5229b4668f067dc 89.4G Yes 103s ago 2024-03-24T11:50:33.878 INFO:teuthology.orchestra.run.smithi130.stdout:smithi174 /dev/nvme2n1 ssd Linux_fe993b0f2ead27ce2e37 89.4G Yes 103s ago 2024-03-24T11:50:33.878 INFO:teuthology.orchestra.run.smithi130.stdout:smithi174 /dev/nvme3n1 ssd Linux_aee018fc9c94715c446f 89.4G Yes 103s ago 2024-03-24T11:50:33.879 INFO:teuthology.orchestra.run.smithi130.stdout:smithi174 /dev/nvme4n1 ssd Linux_b13ba8c559cf545299ce 89.4G No 103s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-24T11:50:33.879 INFO:teuthology.orchestra.run.smithi130.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 79s ago Has a FileSystem, LVM detected 2024-03-24T11:50:33.879 INFO:teuthology.orchestra.run.smithi130.stdout:smithi189 /dev/nvme1n1 ssd Linux_37c0160f32037c6666cb 89.4G Yes 79s ago 2024-03-24T11:50:33.879 INFO:teuthology.orchestra.run.smithi130.stdout:smithi189 /dev/nvme2n1 ssd Linux_6a933c568da872d56090 89.4G Yes 79s ago 2024-03-24T11:50:33.879 INFO:teuthology.orchestra.run.smithi130.stdout:smithi189 /dev/nvme3n1 ssd Linux_f93d6156db02768fcf9a 89.4G Yes 79s ago 2024-03-24T11:50:33.879 INFO:teuthology.orchestra.run.smithi130.stdout:smithi189 /dev/nvme4n1 ssd Linux_60f92fa714c10f261997 89.4G No 79s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-24T11:50:34.523 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-24T11:50:34.762 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:34.787 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:34 smithi130 ceph-mon[25795]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:34.787 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:34 smithi130 ceph-mon[25795]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:34.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:34 smithi189 ceph-mon[30629]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:34.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:34 smithi189 ceph-mon[30629]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:35.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:34 smithi174 ceph-mon[30177]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:35.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:34 smithi174 ceph-mon[30177]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:50:36.637 INFO:teuthology.orchestra.run.smithi130.stdout:service_type: alertmanager 2024-03-24T11:50:36.637 INFO:teuthology.orchestra.run.smithi130.stdout:service_name: alertmanager 2024-03-24T11:50:36.637 INFO:teuthology.orchestra.run.smithi130.stdout:placement: 2024-03-24T11:50:36.637 INFO:teuthology.orchestra.run.smithi130.stdout: count: 1 2024-03-24T11:50:36.637 INFO:teuthology.orchestra.run.smithi130.stdout: hosts: 2024-03-24T11:50:36.637 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi130=a 2024-03-24T11:50:36.637 INFO:teuthology.orchestra.run.smithi130.stdout:status: 2024-03-24T11:50:36.637 INFO:teuthology.orchestra.run.smithi130.stdout: created: '2024-03-24T11:49:30.509931Z' 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout: ports: 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout: - 9093 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout: - 9094 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout: running: 0 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout: size: 1 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout:events: 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout:- 2024-03-24T11:50:17.681544Z service:alertmanager [INFO] "service was created" 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout:--- 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout:service_type: grafana 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout:service_name: grafana 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout:placement: 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout: count: 1 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout: hosts: 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi174=a 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout:spec: 2024-03-24T11:50:36.638 INFO:teuthology.orchestra.run.smithi130.stdout: anonymous_access: true 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout: protocol: https 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout:status: 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout: created: '2024-03-24T11:49:33.283066Z' 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout: ports: 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout: - 3000 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout: running: 0 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout: size: 1 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout:--- 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout:service_type: mgr 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout:service_name: mgr 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout:placement: 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout: count: 3 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout: hosts: 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi130=y 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi174=x 2024-03-24T11:50:36.639 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi189=z 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout:status: 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout: created: '2024-03-24T11:47:55.189833Z' 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout: last_refresh: '2024-03-24T11:49:58.891934Z' 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout: running: 3 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout: size: 3 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout:--- 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout:service_type: mon 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout:service_name: mon 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout:placement: 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout: count: 3 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout: hosts: 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi130:172.21.15.130=a 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi174:172.21.15.174=b 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi189:172.21.15.189=c 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout:status: 2024-03-24T11:50:36.640 INFO:teuthology.orchestra.run.smithi130.stdout: created: '2024-03-24T11:47:27.885928Z' 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout: last_refresh: '2024-03-24T11:49:58.891820Z' 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout: running: 3 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout: size: 3 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout:--- 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout:service_type: node-exporter 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout:service_name: node-exporter 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout:placement: 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout: count: 3 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout: hosts: 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi130=a 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi174=b 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi189=c 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout:status: 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout: created: '2024-03-24T11:49:27.614291Z' 2024-03-24T11:50:36.641 INFO:teuthology.orchestra.run.smithi130.stdout: ports: 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout: - 9100 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout: running: 0 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout: size: 3 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout:--- 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout:service_type: osd 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout:service_name: osd 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout:unmanaged: true 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout:spec: 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout: filter_logic: AND 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout: objectstore: bluestore 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout:status: 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout: container_image_id: 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout: last_refresh: '2024-03-24T11:49:58.892011Z' 2024-03-24T11:50:36.642 INFO:teuthology.orchestra.run.smithi130.stdout: running: 3 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout: size: 3 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout:--- 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout:service_type: prometheus 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout:service_name: prometheus 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout:placement: 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout: count: 1 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout: hosts: 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi174=a 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout:status: 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout: created: '2024-03-24T11:49:24.812750Z' 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout: last_refresh: '2024-03-24T11:49:59.655040Z' 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout: ports: 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout: - 9095 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout: running: 1 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout: size: 1 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout:--- 2024-03-24T11:50:36.643 INFO:teuthology.orchestra.run.smithi130.stdout:service_type: rgw 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout:service_id: foo.a 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout:service_name: rgw.foo.a 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout:placement: 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout: count: 1 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout: hosts: 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout: - smithi130=foo.a 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout:status: 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout: created: '2024-03-24T11:49:25.527757Z' 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout: last_refresh: '2024-03-24T11:50:04.375655Z' 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout: ports: 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout: - 80 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout: running: 1 2024-03-24T11:50:36.644 INFO:teuthology.orchestra.run.smithi130.stdout: size: 1 2024-03-24T11:50:36.843 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:36 smithi174 ceph-mon[30177]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:36.844 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:36 smithi174 systemd[1]: Starting Ceph grafana.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:50:36.906 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:36 smithi130 ceph-mon[25795]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:36.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:36 smithi189 ceph-mon[30629]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:37.225 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:36 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: ::ffff:172.21.15.174 - - [24/Mar/2024:11:50:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:50:37.251 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-24T11:50:37.281 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:36 smithi174 podman[43220]: 2024-03-24 11:50:36.863244351 +0000 UTC m=+0.016982821 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-24T11:50:37.281 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:36 smithi174 podman[43220]: 2024-03-24 11:50:36.987445145 +0000 UTC m=+0.141183616 container create f9e18d02cf2d8ab5bed20c0b134e41c2d56435e2526ce6064e21122f70347d5f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, io.buildah.version=1.32.0, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, architecture=x86_64, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-24T11:50:37.281 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 podman[43220]: 2024-03-24 11:50:37.197763966 +0000 UTC m=+0.351502435 container init f9e18d02cf2d8ab5bed20c0b134e41c2d56435e2526ce6064e21122f70347d5f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a, io.openshift.tags=base rhel8, release=239.1651231664, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, vcs-type=git, io.openshift.expose-services=, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., description=Ceph Grafana Container, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-03-24T11:50:37.281 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 podman[43220]: 2024-03-24 11:50:37.204083805 +0000 UTC m=+0.357822275 container start f9e18d02cf2d8ab5bed20c0b134e41c2d56435e2526ce6064e21122f70347d5f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.expose-services=, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , version=8.5, io.buildah.version=1.32.0, com.redhat.component=ubi8-container) 2024-03-24T11:50:37.281 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 bash[43220]: f9e18d02cf2d8ab5bed20c0b134e41c2d56435e2526ce6064e21122f70347d5f 2024-03-24T11:50:37.491 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:50:37.533 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 systemd[1]: Started Ceph grafana.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:50:37.533 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.35633318Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-24T11:50:37.533 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356532273Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-24T11:50:37.533 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356550822Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-24T11:50:37.533 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356559279Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-24T11:50:37.533 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356565343Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-24T11:50:37.533 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356571103Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-24T11:50:37.533 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356576622Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-24T11:50:37.533 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356582425Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356588183Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356594021Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356600469Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356605792Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356611321Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=settings t=2024-03-24T11:50:37.356616851Z level=info msg="App mode production" 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=sqlstore t=2024-03-24T11:50:37.356662648Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=sqlstore t=2024-03-24T11:50:37.356680043Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.357538807Z level=info msg="Starting DB migrations" 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.357615381Z level=info msg="Executing migration" id="create migration_log table" 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.358141386Z level=info msg="Executing migration" id="create user table" 2024-03-24T11:50:37.534 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.358640406Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.359118401Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.359560837Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.36006102Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.360466586Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.361166145Z level=info msg="Executing migration" id="create user table v2" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.361568872Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.361944499Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.362381806Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.362797028Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.363167739Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.363588287Z level=info msg="Executing migration" id="Update user table charset" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.363867931Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.364297386Z level=info msg="Executing migration" id="Add missing user data" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.364665642Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-24T11:50:37.535 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.365072727Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-24T11:50:37.536 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.365483593Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-24T11:50:37.536 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.365879501Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-24T11:50:37.536 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.366988249Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.372162886Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.372753277Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.375520676Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.378015707Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.378461955Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.381056922Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.383820975Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.386457533Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.388986776Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.391635996Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.394559106Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.397249161Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.399887798Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-24T11:50:37.537 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.402706893Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.405281414Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.407912005Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.408676693Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.409509442Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.410277681Z level=info msg="Executing migration" id="create star table" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.411212711Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.411735337Z level=info msg="Executing migration" id="create org table v1" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.412669284Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.413387912Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.413828192Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.414655049Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.415490363Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.415903833Z level=info msg="Executing migration" id="Update org table charset" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.416229714Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-24T11:50:37.538 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.417144651Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.417853111Z level=info msg="Executing migration" id="create dashboard table" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.418636146Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.419087588Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.419919604Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.420736953Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.421551878Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.422000614Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.422871484Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.423733932Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.424585034Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.425063279Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.425950309Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.426711666Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.427589766Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-24T11:50:37.539 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.428259395Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.428781464Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.42961975Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.430505691Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.43139521Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.431829894Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.432831363Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.433573027Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.434419178Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.434989965Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.435547284Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.438175191Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.438702296Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.440794473Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.441284533Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.441686447Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-24T11:50:37.540 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.44465987Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.445236188Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.447644698Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.450324885Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.450759847Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.452619152Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.454750807Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.456621751Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.457167615Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.458978619Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.46087716Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.462744027Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.463075441Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.465141052Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.466904837Z level=info msg="Executing migration" id="create data_source table" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.468818724Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-24T11:50:37.541 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.469280293Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.471322049Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.473123063Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.474971803Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.47600635Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.478003723Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.479922653Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.482088081Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.482486832Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.484372643Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.484929841Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.486241916Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-24T11:50:37.542 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.488193582Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.488782726Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.490642342Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.490978743Z level=info msg="Executing migration" id="Add uid column" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.492711673Z level=info msg="Executing migration" id="Update uid value" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.494487056Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.494947298Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.496853141Z level=info msg="Executing migration" id="create api_key table" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.498808365Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.500775338Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.501217472Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.503268039Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.505305403Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.50731609Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.509214175Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-24T11:50:37.543 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.51102623Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-24T11:50:37.544 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.513035282Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-24T11:50:37.544 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.514824005Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-24T11:50:37.544 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.516852987Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-24T11:50:37.544 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.521155506Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-24T11:50:37.544 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.525036463Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-24T11:50:37.544 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.528944933Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-24T11:50:37.544 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.532934459Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-24T11:50:37.785 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-mon[30177]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-24T11:50:37.785 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:37.785 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:37.785 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:37.785 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-mon[30177]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:37.786 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.536598285Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-24T11:50:37.786 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.540577659Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-24T11:50:37.786 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.544421089Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-24T11:50:37.786 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.548304512Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-24T11:50:37.786 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.552259449Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.556272128Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.560230655Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.564614608Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.568342762Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.572336683Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.576092626Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.580126122Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.584146992Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.587971308Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.589992439Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.591871269Z level=info msg="Executing migration" id="create quota table v1" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.593842152Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-24T11:50:37.787 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.595806831Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.597689632Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.599478706Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.601327625Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.603184699Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.605172406Z level=info msg="Executing migration" id="create session table" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.60690028Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.609111664Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.611071909Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.612936902Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.614891474Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.616543119Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.61857165Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.620414408Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-24T11:50:37.788 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.622352931Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.624229944Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.626070724Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.628005271Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.629864026Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.631725225Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.633613799Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.635612293Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.637478256Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.639273625Z level=info msg="Executing migration" id="create alert table v1" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.641330122Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.643347189Z level=info msg="Executing migration" id="add index alert state" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.645355657Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.647112403Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-24T11:50:37.789 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.649144486Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.650951079Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.652925624Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.656632587Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.660537127Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.661057559Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.665007092Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.668943705Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.669341159Z level=info msg="Executing migration" id="Add column is_default" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.67326287Z level=info msg="Executing migration" id="Add column frequency" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.673895664Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.677487649Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.681426932Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-24T11:50:37.790 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.685526099Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.689782019Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.693718722Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.697582355Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.701580738Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.705470364Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.709417721Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.713396709Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.715550409Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.717664534Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.719713969Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.721813931Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.723879657Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.72603467Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.728051373Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-24T11:50:37.791 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.729818638Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-24T11:50:37.792 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.731803315Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-24T11:50:37.792 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.73368038Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-24T11:50:37.792 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.734180753Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-24T11:50:37.792 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.735291031Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-24T11:50:37.792 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.735913188Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-24T11:50:37.792 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.736770516Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-24T11:50:37.792 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.73784054Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-24T11:50:37.792 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.738539372Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-24T11:50:37.792 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.739525083Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.740324447Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.741316792Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.744029509Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.74681974Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.749480898Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.752293834Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.755053141Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.757864993Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.760732399Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.763513615Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.766368024Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.769266882Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-24T11:50:37.793 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.771971066Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-24T11:50:37.794 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.774743034Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-24T11:50:37.794 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.77726166Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-24T11:50:37.794 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.780004775Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-24T11:50:37.794 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.782560491Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-24T11:50:37.794 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.78542683Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-24T11:50:37.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:37 smithi189 ceph-mon[30629]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-24T11:50:37.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:37 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:37.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:37 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:37.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:37 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:37.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:37 smithi189 ceph-mon[30629]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:37.946 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:37 smithi130 ceph-mon[25795]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-24T11:50:37.946 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:37 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:37.946 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:37 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:37.946 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:37 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:37.946 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:37 smithi130 ceph-mon[25795]: from='mgr.14448 172.21.15.130:0/839730351' entity='mgr.y' 2024-03-24T11:50:38.036 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.788029564Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-24T11:50:38.036 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.790804103Z level=info msg="Executing migration" id="Move region to single row" 2024-03-24T11:50:38.036 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.793277641Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-24T11:50:38.036 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.795900992Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-24T11:50:38.036 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.798481775Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-24T11:50:38.036 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.801275605Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-24T11:50:38.036 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.803935803Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-24T11:50:38.036 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.80636559Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-24T11:50:38.036 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.809079365Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-24T11:50:38.036 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.811587426Z level=info msg="Executing migration" id="create test_data table" 2024-03-24T11:50:38.036 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.812191131Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-24T11:50:38.037 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.814112654Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-24T11:50:38.037 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.816782111Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-24T11:50:38.037 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.819445022Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-24T11:50:38.037 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.82214656Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-24T11:50:38.037 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.824698253Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-24T11:50:38.037 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.827431967Z level=info msg="Executing migration" id="create team table" 2024-03-24T11:50:38.037 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.830116706Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-24T11:50:38.038 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.832984014Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-24T11:50:38.038 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.835628649Z level=info msg="Executing migration" id="create team member table" 2024-03-24T11:50:38.038 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.836286511Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-24T11:50:38.038 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.837099482Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-24T11:50:38.038 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.837965335Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-24T11:50:38.038 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.838713296Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-24T11:50:38.038 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.839490142Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-24T11:50:38.038 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.840311565Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-24T11:50:38.038 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.841172184Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.842024366Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.84287318Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.84366311Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.844080948Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.844917848Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.845753156Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.846381823Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.846796409Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.847195338Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.84748398Z level=info msg="Executing migration" id="create tag table" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.847854717Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.848278792Z level=info msg="Executing migration" id="create login attempt table" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.848648535Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.849058969Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-24T11:50:38.039 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.849454704Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.851292109Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.851670227Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.852078015Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.852385557Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.852710188Z level=info msg="Executing migration" id="create user auth table" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.853053494Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.853459299Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.853716004Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.854471072Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.855236022Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.855961549Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.856736497Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.857155905Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.857870875Z level=info msg="Executing migration" id="create server_lock table" 2024-03-24T11:50:38.040 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.858256913Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.858675392Z level=info msg="Executing migration" id="create user auth token table" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.859102561Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.85985847Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.860311523Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.860750624Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.861539066Z level=info msg="Executing migration" id="create cache_data table" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.861962896Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.862412677Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.862858394Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.863731325Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.864276596Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.864575964Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.865018522Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.865882386Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-24T11:50:38.041 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.866658896Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.866956086Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.867476899Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.868493301Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.869166546Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.870022366Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.870818814Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.871800406Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.872532321Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.873525301Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.874246987Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.875194735Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.875880792Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.876756891Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-24T11:50:38.042 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.877210462Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.878175126Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.87898699Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.879851859Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.880306129Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.881191673Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.884609223Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.887950718Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.88842604Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.891128687Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.891966096Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.894143096Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.896888469Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.899517561Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.902429519Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-24T11:50:38.043 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.905126505Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.907853133Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.910741436Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.913568489Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.916294827Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.918986614Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.919936657Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.920782162Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.921778627Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.922702115Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.923562736Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.924568798Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.925268265Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.926144311Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.926965326Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.927808213Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.928770099Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-24T11:50:38.044 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.9296489Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-24T11:50:38.045 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.93052285Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-24T11:50:38.045 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.931413359Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-24T11:50:38.045 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.932304659Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-24T11:50:38.045 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.93326733Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-24T11:50:38.045 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.934002462Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-24T11:50:38.045 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.934887555Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-24T11:50:38.045 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.935883393Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-24T11:50:38.045 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.936822703Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-24T11:50:38.045 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.93777533Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-24T11:50:38.045 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.938822939Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-24T11:50:38.045 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.939689601Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-24T11:50:38.046 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.940665459Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-24T11:50:38.046 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.941612635Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-24T11:50:38.046 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.94248108Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-24T11:50:38.046 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.943313881Z level=info msg="Executing migration" id="create alert_image table" 2024-03-24T11:50:38.046 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.944163781Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-24T11:50:38.046 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.945048066Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-24T11:50:38.046 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.945998438Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-24T11:50:38.046 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.946896335Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.947621401Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.947725173Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.948515486Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.949054228Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.949505516Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.949640204Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.94977136Z level=info msg="alerts found to migrate" alerts=0 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.952463192Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.955428953Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.958324007Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.961297976Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.964241852Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.96684915Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.969778296Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-24T11:50:38.047 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.972876729Z level=info msg="Executing migration" id="create data_keys table" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.975682264Z level=info msg="Executing migration" id="create secrets table" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.97870381Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.982974999Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.98578524Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.988755914Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.993043747Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:37 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:37.997152649Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.000065335Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.003139986Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.006046818Z level=info msg="Executing migration" id="create permission table" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.008803481Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.011599025Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.014299312Z level=info msg="Executing migration" id="create role table" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.017132978Z level=info msg="Executing migration" id="add column display_name" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.019739082Z level=info msg="Executing migration" id="add column group_name" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.022661663Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-24T11:50:38.048 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.025374421Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-24T11:50:38.049 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.028113055Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-24T11:50:38.049 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.030731667Z level=info msg="Executing migration" id="create team role table" 2024-03-24T11:50:38.049 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.033633279Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-24T11:50:38.049 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.03627083Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-24T11:50:38.049 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.039068311Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-24T11:50:38.344 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.041775492Z level=info msg="Executing migration" id="create user role table" 2024-03-24T11:50:38.344 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.044453402Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-24T11:50:38.344 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.047204849Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-24T11:50:38.344 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.050093324Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.050962966Z level=info msg="Executing migration" id="create builtin role table" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.051730569Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.052703712Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.053421648Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.054469467Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.055337947Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.056170202Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.057144546Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.05790412Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.058851652Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.05958604Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.060540253Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.061394513Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-24T11:50:38.345 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.062244083Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-24T11:50:38.346 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.063326829Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-24T11:50:38.346 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.06406125Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-24T11:50:38.346 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.064812651Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-24T11:50:38.346 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.065675703Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-24T11:50:38.346 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.066680226Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.067419981Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.068267143Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.069226926Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.070323903Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.071279526Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.07220231Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.072890003Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.073860897Z level=info msg="Executing migration" id="add correlation config column" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.074958849Z level=info msg="Executing migration" id="create entity_events table" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.075843879Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.076691785Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.07680235Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.07753516Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.0776373Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:50:38.347 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.078387709Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.079221716Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.080239141Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.080923877Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.083911818Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.086764248Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.092449271Z level=info msg="Executing migration" id="Drop public config table" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.095369081Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.09833459Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.101255345Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.105079582Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.107827718Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.110841853Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.113663371Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-24T11:50:38.348 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.116682934Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.119579436Z level=info msg="Executing migration" id="add share column" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.122360906Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.125248193Z level=info msg="Executing migration" id="create file table" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.127983157Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.128447037Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.131259618Z level=info msg="Executing migration" id="create file_meta table" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.13400219Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.136831251Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.137167531Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.139996633Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.142711227Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.145499044Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.14672606Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.149569382Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-24T11:50:38.349 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.152099514Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-24T11:50:38.350 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.154767711Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-24T11:50:38.350 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.155251971Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-24T11:50:38.350 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.157876044Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-24T11:50:38.350 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.160485186Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-24T11:50:38.350 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.162882395Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-24T11:50:38.350 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.163336417Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-24T11:50:38.350 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.173055635Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-24T11:50:38.350 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.175835573Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.178642487Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.182058826Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.184786517Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.187368549Z level=info msg="Executing migration" id="create folder table" 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.188118017Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.188558085Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.189377499Z level=info msg="Executing migration" id="Update folder title length" 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.190254411Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=migrator t=2024-03-24T11:50:38.190921223Z level=info msg="migrations completed" performed=485 skipped=0 duration=833.329555ms 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=sqlstore t=2024-03-24T11:50:38.191324503Z level=info msg="Created default organization" 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=plugin.loader t=2024-03-24T11:50:38.246623518Z level=info msg="Plugin registered" pluginID=input 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=plugin.loader t=2024-03-24T11:50:38.274611313Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=plugin.loader t=2024-03-24T11:50:38.274643874Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-24T11:50:38.351 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=secrets t=2024-03-24T11:50:38.274678306Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-24T11:50:38.352 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=query_data t=2024-03-24T11:50:38.276862604Z level=info msg="Query Service initialization" 2024-03-24T11:50:38.352 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=live.push_http t=2024-03-24T11:50:38.279287155Z level=info msg="Live Push Gateway initialization" 2024-03-24T11:50:38.919 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-mon[30177]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:38.919 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-mon[30177]: Deploying daemon node-exporter.a on smithi130 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=infra.usagestats.collector t=2024-03-24T11:50:38.880752219Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=provisioning.datasources t=2024-03-24T11:50:38.881760425Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=provisioning.datasources t=2024-03-24T11:50:38.892776364Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=provisioning.alerting t=2024-03-24T11:50:38.903793454Z level=info msg="starting to provision alerting" 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=provisioning.alerting t=2024-03-24T11:50:38.903812144Z level=info msg="finished to provision alerting" 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=ngalert.state.manager t=2024-03-24T11:50:38.904089369Z level=info msg="Warming state cache for startup" 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=grafanaStorageLogger t=2024-03-24T11:50:38.904353885Z level=info msg="storage starting" 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=http.server t=2024-03-24T11:50:38.905785418Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=ngalert.state.manager t=2024-03-24T11:50:38.906351878Z level=info msg="State cache has been initialized" states=0 duration=2.2621ms 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=ticker t=2024-03-24T11:50:38.906417487Z level=info msg=starting first_tick=2024-03-24T11:50:40Z 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=ngalert.multiorg.alertmanager t=2024-03-24T11:50:38.906418594Z level=info msg="starting MultiOrg Alertmanager" 2024-03-24T11:50:38.920 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=sqlstore.transactions t=2024-03-24T11:50:38.916710242Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:50:38.921 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=sqlstore.transactions t=2024-03-24T11:50:38.919929948Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:50:38.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:38 smithi189 ceph-mon[30629]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:38.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:38 smithi189 ceph-mon[30629]: Deploying daemon node-exporter.a on smithi130 2024-03-24T11:50:38.972 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:38 smithi130 ceph-mon[25795]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:50:38.972 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:38 smithi130 ceph-mon[25795]: Deploying daemon node-exporter.a on smithi130 2024-03-24T11:50:39.343 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=sqlstore.transactions t=2024-03-24T11:50:38.93041142Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-24T11:50:39.344 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:50:38 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a[43234]: logger=sqlstore.transactions t=2024-03-24T11:50:38.941172512Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-24T11:50:39.460 INFO:teuthology.orchestra.run.smithi130.stdout:osd 3 40s ago - 2024-03-24T11:50:40.038 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:39 smithi130 bash[56476]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-24T11:50:40.161 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T11:50:40.170 INFO:tasks.cephadm:Teardown begin 2024-03-24T11:50:40.170 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:40.200 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:40.238 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:50:40.275 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T11:50:40.276 DEBUG:teuthology.orchestra.run.smithi130:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:50:40.290 DEBUG:teuthology.orchestra.run.smithi174:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:50:40.306 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:50:40.332 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T11:50:40.332 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T11:50:40.332 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.a 2024-03-24T11:50:40.537 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:40 smithi130 systemd[1]: Stopping Ceph mon.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:50:41.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:40 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a[25772]: 2024-03-24T11:50:40.667+0000 7f0d9439a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:41.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:40 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a[25772]: 2024-03-24T11:50:40.667+0000 7f0d9439a700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T11:50:41.038 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:40 smithi130 podman[56591]: 2024-03-24 11:50:40.760181829 +0000 UTC m=+0.242985921 container died 360e2bed32ee5dbce3791bad8e60bef7b16b0d536660e8bdaf6785e4f5e4c7c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T11:50:41.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:40 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:40] ENGINE Bus STOPPING 2024-03-24T11:50:41.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:40 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:50:41.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:40 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:40] ENGINE Bus STOPPED 2024-03-24T11:50:41.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:40 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:40] ENGINE Bus STARTING 2024-03-24T11:50:41.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:40 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:40] ENGINE Serving on http://:::9283 2024-03-24T11:50:41.038 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:40 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:40] ENGINE Bus STARTED 2024-03-24T11:50:41.537 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:41 smithi130 podman[56591]: 2024-03-24 11:50:41.41944199 +0000 UTC m=+0.902246083 container cleanup 360e2bed32ee5dbce3791bad8e60bef7b16b0d536660e8bdaf6785e4f5e4c7c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T11:50:41.537 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:41 smithi130 bash[56591]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a 2024-03-24T11:50:42.037 INFO:journalctl@ceph.mon.a.smithi130.stdout:Mar 24 11:50:41 smithi130 podman[56627]: 2024-03-24 11:50:41.72993181 +0000 UTC m=+0.965925420 container remove 360e2bed32ee5dbce3791bad8e60bef7b16b0d536660e8bdaf6785e4f5e4c7c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T11:50:42.071 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T11:50:42.130 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:42.130 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T11:50:42.131 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-24T11:50:42.131 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.b 2024-03-24T11:50:42.332 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 bash[56476]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T11:50:42.332 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 bash[56476]: Writing manifest to image destination 2024-03-24T11:50:42.420 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:42 smithi174 systemd[1]: Stopping Ceph mon.b for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:50:42.634 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:42] ENGINE Bus STOPPING 2024-03-24T11:50:42.634 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:50:42.635 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:42] ENGINE Bus STOPPED 2024-03-24T11:50:42.635 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:42] ENGINE Bus STARTING 2024-03-24T11:50:42.635 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:42] ENGINE Serving on http://:::9283 2024-03-24T11:50:42.635 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: [24/Mar/2024:11:50:42] ENGINE Bus STARTED 2024-03-24T11:50:42.818 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:42 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-b[30154]: 2024-03-24T11:50:42.420+0000 7f6aed407700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:42.818 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:42 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-b[30154]: 2024-03-24T11:50:42.420+0000 7f6aed407700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T11:50:42.819 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:42 smithi174 podman[43356]: 2024-03-24 11:50:42.543701728 +0000 UTC m=+0.226284672 container died fa9693ddc64f06f827549c5ab3fbe214355ef8ee8c9314c457235cdfa818c2ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T11:50:42.955 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 podman[56476]: 2024-03-24 11:50:42.61316818 +0000 UTC m=+3.305266448 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T11:50:42.956 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 podman[56476]: 2024-03-24 11:50:42.728709309 +0000 UTC m=+3.420807572 container create 360cf4dfbc7612d128eeab808530f25599972b56b4649071295ebedead16dbc1 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:50:42.956 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 podman[56476]: 2024-03-24 11:50:42.955958061 +0000 UTC m=+3.648056326 container init 360cf4dfbc7612d128eeab808530f25599972b56b4649071295ebedead16dbc1 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:50:43.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:42 smithi174 podman[43356]: 2024-03-24 11:50:42.909710605 +0000 UTC m=+0.592293600 container cleanup fa9693ddc64f06f827549c5ab3fbe214355ef8ee8c9314c457235cdfa818c2ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-b, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T11:50:43.093 INFO:journalctl@ceph.mon.b.smithi174.stdout:Mar 24 11:50:42 smithi174 bash[43356]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-b 2024-03-24T11:50:43.288 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 podman[56476]: 2024-03-24 11:50:42.961023376 +0000 UTC m=+3.653121641 container start 360cf4dfbc7612d128eeab808530f25599972b56b4649071295ebedead16dbc1 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:50:43.288 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.964Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T11:50:43.288 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.964Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T11:50:43.288 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T11:50:43.288 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T11:50:43.289 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T11:50:43.290 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T11:50:43.290 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T11:50:43.290 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T11:50:43.290 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T11:50:43.290 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T11:50:43.290 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T11:50:43.290 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T11:50:43.290 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T11:50:43.290 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T11:50:43.290 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T11:50:43.291 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.965Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.966Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.966Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:42 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a[56736]: ts=2024-03-24T11:50:42.966Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:43 smithi130 bash[56476]: 360cf4dfbc7612d128eeab808530f25599972b56b4649071295ebedead16dbc1 2024-03-24T11:50:43.292 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:50:43 smithi130 systemd[1]: Started Ceph node-exporter.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:50:43.519 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T11:50:43.575 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:43.575 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-24T11:50:43.575 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-24T11:50:43.576 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.c 2024-03-24T11:50:43.900 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:43 smithi189 systemd[1]: Stopping Ceph mon.c for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:50:44.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:43 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-c[30597]: 2024-03-24T11:50:43.898+0000 7f9d454c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:44.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:43 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-c[30597]: 2024-03-24T11:50:43.898+0000 7f9d454c0700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T11:50:44.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:44 smithi189 podman[43774]: 2024-03-24 11:50:44.083646906 +0000 UTC m=+0.316458992 container died 9df70d4d8c048f4be99decb4ba739fd6293d7769dc20ff38b9f1c61ebbc134d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T11:50:44.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:44 smithi189 podman[43774]: 2024-03-24 11:50:44.554778538 +0000 UTC m=+0.787590693 container cleanup 9df70d4d8c048f4be99decb4ba739fd6293d7769dc20ff38b9f1c61ebbc134d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-db0330b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:50:44.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:44 smithi189 bash[43774]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-c 2024-03-24T11:50:44.971 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T11:50:44.995 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:44 smithi189 podman[43791]: 2024-03-24 11:50:44.744140945 +0000 UTC m=+0.656733836 container remove 9df70d4d8c048f4be99decb4ba739fd6293d7769dc20ff38b9f1c61ebbc134d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T11:50:44.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:44 smithi189 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-24T11:50:44.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:44 smithi189 systemd[1]: Stopped Ceph mon.c for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:50:44.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:50:44 smithi189 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mon.c.service: Consumed 3.521s CPU time. 2024-03-24T11:50:45.021 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:45.021 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-24T11:50:45.021 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T11:50:45.021 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.y 2024-03-24T11:50:45.329 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:45 smithi130 systemd[1]: Stopping Ceph mgr.y for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:50:45.609 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:45 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y[26054]: 2024-03-24T11:50:45.328+0000 7f4b711be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:45.610 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:45 smithi130 podman[56814]: 2024-03-24 11:50:45.361056512 +0000 UTC m=+0.141263803 container died 44df44b989736a7c6332e02d896794ab6ecb01238b9aad8d76fecc6c54ab51ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T11:50:45.610 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:45 smithi130 podman[56814]: 2024-03-24 11:50:45.609847389 +0000 UTC m=+0.390054680 container cleanup 44df44b989736a7c6332e02d896794ab6ecb01238b9aad8d76fecc6c54ab51ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T11:50:45.926 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:45 smithi130 bash[56814]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y 2024-03-24T11:50:45.926 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Mar 24 11:50:45 smithi130 podman[56833]: 2024-03-24 11:50:45.74455222 +0000 UTC m=+0.386133099 container remove 44df44b989736a7c6332e02d896794ab6ecb01238b9aad8d76fecc6c54ab51ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T11:50:45.940 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T11:50:45.975 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:45.975 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T11:50:45.975 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T11:50:45.976 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.x 2024-03-24T11:50:46.343 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:50:46 smithi174 systemd[1]: Stopping Ceph mgr.x for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:50:46.647 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:50:46 smithi174 podman[43487]: 2024-03-24 11:50:46.38705059 +0000 UTC m=+0.220480716 container died 3909890b4ca3e23dba86ba0cc4e98a579ffd1735add610c8b1d33cf58caa9fa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T11:50:47.039 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:50:46 smithi174 podman[43487]: 2024-03-24 11:50:46.771216517 +0000 UTC m=+0.604646624 container cleanup 3909890b4ca3e23dba86ba0cc4e98a579ffd1735add610c8b1d33cf58caa9fa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T11:50:47.039 INFO:journalctl@ceph.mgr.x.smithi174.stdout:Mar 24 11:50:46 smithi174 bash[43487]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-x 2024-03-24T11:50:47.136 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T11:50:47.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:47.173 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T11:50:47.173 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-24T11:50:47.173 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.z 2024-03-24T11:50:47.606 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:50:47 smithi189 systemd[1]: Stopping Ceph mgr.z for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:50:47.908 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:50:47 smithi189 podman[43905]: 2024-03-24 11:50:47.607173362 +0000 UTC m=+0.242887599 container died 78ee45e0c014da72a244eb0dfb735f24f197d0ff3e90cc4bea2c3c0f2137e5e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-03-24T11:50:48.195 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:50:47 smithi189 podman[43905]: 2024-03-24 11:50:47.983318078 +0000 UTC m=+0.619032322 container cleanup 78ee45e0c014da72a244eb0dfb735f24f197d0ff3e90cc4bea2c3c0f2137e5e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:50:48.195 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:50:47 smithi189 bash[43905]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-mgr-z 2024-03-24T11:50:48.390 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-24T11:50:48.425 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:48.425 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-24T11:50:48.425 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T11:50:48.425 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@osd.0 2024-03-24T11:50:48.708 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:48 smithi130 systemd[1]: Stopping Ceph osd.0 for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:50:49.037 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:48 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0[40578]: 2024-03-24T11:50:48.707+0000 7fb32b25a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:49.037 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:48 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0[40578]: 2024-03-24T11:50:48.707+0000 7fb32b25a700 -1 osd.0 32 *** Got signal Terminated *** 2024-03-24T11:50:49.038 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:48 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0[40578]: 2024-03-24T11:50:48.707+0000 7fb32b25a700 -1 osd.0 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:50:49.094 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:50:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:50:48.828Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.130:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.130:8765: connect: connection refused" 2024-03-24T11:50:49.094 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:50:48 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:50:48.828Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.130:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.130:8765: connect: connection refused" 2024-03-24T11:50:54.443 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:54 smithi130 podman[56945]: 2024-03-24 11:50:54.165011656 +0000 UTC m=+5.546191000 container died 77c04ee664658fc55711a97239591c9d8a1f1b0809a6224af47edff476677ba5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T11:50:54.781 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:54 smithi130 podman[56945]: 2024-03-24 11:50:54.555304742 +0000 UTC m=+5.936484095 container cleanup 77c04ee664658fc55711a97239591c9d8a1f1b0809a6224af47edff476677ba5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T11:50:54.781 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:54 smithi130 bash[56945]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0 2024-03-24T11:50:55.037 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:54 smithi130 podman[56969]: 2024-03-24 11:50:54.781603268 +0000 UTC m=+0.610456888 container remove 77c04ee664658fc55711a97239591c9d8a1f1b0809a6224af47edff476677ba5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T11:50:55.538 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:55 smithi130 podman[57068]: 2024-03-24 11:50:54.999688335 +0000 UTC m=+0.017612745 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:50:55.538 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:55 smithi130 podman[57068]: 2024-03-24 11:50:55.098959854 +0000 UTC m=+0.116884259 container create c954f5c926828ea7638795bcbe524dd5bc7d5e547ffc143230a1d2e9818edcf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:50:55.538 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:55 smithi130 podman[57068]: 2024-03-24 11:50:55.251383154 +0000 UTC m=+0.269307571 container init c954f5c926828ea7638795bcbe524dd5bc7d5e547ffc143230a1d2e9818edcf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:50:55.538 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:55 smithi130 podman[57068]: 2024-03-24 11:50:55.256415807 +0000 UTC m=+0.274340220 container start c954f5c926828ea7638795bcbe524dd5bc7d5e547ffc143230a1d2e9818edcf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T11:50:55.539 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:55 smithi130 podman[57068]: 2024-03-24 11:50:55.342764887 +0000 UTC m=+0.360689301 container attach c954f5c926828ea7638795bcbe524dd5bc7d5e547ffc143230a1d2e9818edcf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T11:50:55.845 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:55 smithi130 podman[57068]: 2024-03-24 11:50:55.547052426 +0000 UTC m=+0.564976832 container died c954f5c926828ea7638795bcbe524dd5bc7d5e547ffc143230a1d2e9818edcf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T11:50:56.084 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T11:50:56.110 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:56 smithi130 podman[57114]: 2024-03-24 11:50:56.062965604 +0000 UTC m=+0.506918606 container remove c954f5c926828ea7638795bcbe524dd5bc7d5e547ffc143230a1d2e9818edcf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-24T11:50:56.111 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:56 smithi130 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-24T11:50:56.111 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:56 smithi130 systemd[1]: Stopped Ceph osd.0 for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:50:56.111 INFO:journalctl@ceph.osd.0.smithi130.stdout:Mar 24 11:50:56 smithi130 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@osd.0.service: Consumed 3.686s CPU time. 2024-03-24T11:50:56.126 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:50:56.127 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T11:50:56.127 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T11:50:56.127 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@osd.1 2024-03-24T11:50:56.433 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:50:56 smithi174 systemd[1]: Stopping Ceph osd.1 for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:50:56.843 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:50:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1[34749]: 2024-03-24T11:50:56.432+0000 7ffad8e6f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:50:56.844 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:50:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1[34749]: 2024-03-24T11:50:56.432+0000 7ffad8e6f700 -1 osd.1 32 *** Got signal Terminated *** 2024-03-24T11:50:56.844 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:50:56 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1[34749]: 2024-03-24T11:50:56.432+0000 7ffad8e6f700 -1 osd.1 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:51:02.022 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:51:01 smithi174 podman[43622]: 2024-03-24 11:51:01.751395568 +0000 UTC m=+5.431344905 container died 98ba3d93580cee0c4629776591540cb0c8682048eaaf637f9b47cd544a1f569e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-24T11:51:02.343 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:51:02 smithi174 podman[43622]: 2024-03-24 11:51:02.137594314 +0000 UTC m=+5.817543650 container cleanup 98ba3d93580cee0c4629776591540cb0c8682048eaaf637f9b47cd544a1f569e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:51:02.343 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:51:02 smithi174 bash[43622]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1 2024-03-24T11:51:02.691 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:51:02 smithi174 podman[43644]: 2024-03-24 11:51:02.37178065 +0000 UTC m=+0.621600027 container remove 98ba3d93580cee0c4629776591540cb0c8682048eaaf637f9b47cd544a1f569e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T11:51:03.000 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:51:02 smithi174 podman[43739]: 2024-03-24 11:51:02.59464274 +0000 UTC m=+0.017852422 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:51:03.000 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:51:02 smithi174 podman[43739]: 2024-03-24 11:51:02.705833017 +0000 UTC m=+0.129042675 container create 87abd1ffd7bb3815c469a3fde001de988cb50bb8be6a6e8484522394146be153 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T11:51:03.000 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:51:02 smithi174 podman[43739]: 2024-03-24 11:51:02.916531016 +0000 UTC m=+0.339740673 container init 87abd1ffd7bb3815c469a3fde001de988cb50bb8be6a6e8484522394146be153 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-03-24T11:51:03.000 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:51:02 smithi174 podman[43739]: 2024-03-24 11:51:02.922195484 +0000 UTC m=+0.345405145 container start 87abd1ffd7bb3815c469a3fde001de988cb50bb8be6a6e8484522394146be153 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T11:51:03.000 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:51:03 smithi174 podman[43739]: 2024-03-24 11:51:02.999678234 +0000 UTC m=+0.422887902 container attach 87abd1ffd7bb3815c469a3fde001de988cb50bb8be6a6e8484522394146be153 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:51:03.343 INFO:journalctl@ceph.osd.1.smithi174.stdout:Mar 24 11:51:03 smithi174 podman[43739]: 2024-03-24 11:51:03.213595949 +0000 UTC m=+0.636805615 container died 87abd1ffd7bb3815c469a3fde001de988cb50bb8be6a6e8484522394146be153 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T11:51:03.838 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T11:51:03.878 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:51:03.878 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T11:51:03.878 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T11:51:03.878 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@osd.2 2024-03-24T11:51:04.189 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:03 smithi189 systemd[1]: Stopping Ceph osd.2 for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:51:04.445 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:04 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2[35929]: 2024-03-24T11:51:04.188+0000 7f5df0ae1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:51:04.445 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:04 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2[35929]: 2024-03-24T11:51:04.188+0000 7f5df0ae1700 -1 osd.2 32 *** Got signal Terminated *** 2024-03-24T11:51:04.445 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:04 smithi189 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2[35929]: 2024-03-24T11:51:04.188+0000 7f5df0ae1700 -1 osd.2 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:51:09.827 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:09 smithi189 podman[44038]: 2024-03-24 11:51:09.57740651 +0000 UTC m=+5.498976556 container died dd47e0f1ecaf5c80aece05482236dceb1a8cba1abdd7fe7bd0434613bfc6b189 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T11:51:10.195 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:09 smithi189 podman[44038]: 2024-03-24 11:51:09.926382018 +0000 UTC m=+5.847952052 container cleanup dd47e0f1ecaf5c80aece05482236dceb1a8cba1abdd7fe7bd0434613bfc6b189 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T11:51:10.195 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:09 smithi189 bash[44038]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2 2024-03-24T11:51:10.549 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:10 smithi189 podman[44071]: 2024-03-24 11:51:10.202154127 +0000 UTC m=+0.621843578 container remove dd47e0f1ecaf5c80aece05482236dceb1a8cba1abdd7fe7bd0434613bfc6b189 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:51:10.813 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:10 smithi189 podman[44168]: 2024-03-24 11:51:10.45392707 +0000 UTC m=+0.019185992 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:51:10.813 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:10 smithi189 podman[44168]: 2024-03-24 11:51:10.561409683 +0000 UTC m=+0.126668606 container create f0a4c92a6b7622780945b57829d7057a2358eb822ec49e804d05a24b8148a2cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T11:51:10.814 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:10 smithi189 podman[44168]: 2024-03-24 11:51:10.747266573 +0000 UTC m=+0.312525505 container init f0a4c92a6b7622780945b57829d7057a2358eb822ec49e804d05a24b8148a2cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:51:10.814 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:10 smithi189 podman[44168]: 2024-03-24 11:51:10.752894324 +0000 UTC m=+0.318153243 container start f0a4c92a6b7622780945b57829d7057a2358eb822ec49e804d05a24b8148a2cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-db0330b) 2024-03-24T11:51:10.814 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:10 smithi189 podman[44168]: 2024-03-24 11:51:10.813621783 +0000 UTC m=+0.378880711 container attach f0a4c92a6b7622780945b57829d7057a2358eb822ec49e804d05a24b8148a2cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T11:51:11.195 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:11 smithi189 podman[44168]: 2024-03-24 11:51:11.052942712 +0000 UTC m=+0.618201630 container died f0a4c92a6b7622780945b57829d7057a2358eb822ec49e804d05a24b8148a2cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b) 2024-03-24T11:51:11.583 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:51:11 smithi189 podman[44214]: 2024-03-24 11:51:11.582961247 +0000 UTC m=+0.525216015 container remove f0a4c92a6b7622780945b57829d7057a2358eb822ec49e804d05a24b8148a2cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T11:51:11.608 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T11:51:11.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:51:11.647 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T11:51:11.647 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-24T11:51:11.647 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@rgw.foo.a 2024-03-24T11:51:11.958 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:51:11 smithi130 systemd[1]: Stopping Ceph rgw.foo.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:51:12.287 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:51:11 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-rgw-foo-a[47151]: 2024-03-24T11:51:11.958+0000 7f3042da0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:51:12.287 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:51:11 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-rgw-foo-a[47151]: 2024-03-24T11:51:11.958+0000 7f3051610ac0 -1 shutting down 2024-03-24T11:51:22.288 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:51:21 smithi130 bash[57214]: time="2024-03-24T11:51:21Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-24T11:51:22.288 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:51:21 smithi130 podman[57214]: 2024-03-24 11:51:21.978548673 +0000 UTC m=+10.130420629 container died b58dd87162303f1cfe96a59abb07393a98e2dfeed7b148a7cbac2267c5f76075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T11:51:22.710 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:51:22 smithi130 podman[57214]: 2024-03-24 11:51:22.354684007 +0000 UTC m=+10.506555963 container cleanup b58dd87162303f1cfe96a59abb07393a98e2dfeed7b148a7cbac2267c5f76075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T11:51:22.710 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:51:22 smithi130 bash[57214]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-rgw-foo-a 2024-03-24T11:51:22.710 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:51:22 smithi130 podman[57239]: 2024-03-24 11:51:22.497328278 +0000 UTC m=+0.513149042 container remove b58dd87162303f1cfe96a59abb07393a98e2dfeed7b148a7cbac2267c5f76075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T11:51:22.710 INFO:journalctl@ceph.rgw.foo.a.smithi130.stdout:Mar 24 11:51:22 smithi130 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-24T11:51:22.727 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@rgw.foo.a.service' 2024-03-24T11:51:22.764 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:51:22.765 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-24T11:51:22.765 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-24T11:51:22.765 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@prometheus.a 2024-03-24T11:51:23.058 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:22 smithi174 systemd[1]: Stopping Ceph prometheus.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:51:23.343 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:51:23.344 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T11:51:23.344 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T11:51:23.344 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T11:51:23.344 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T11:51:23.344 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T11:51:23.344 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T11:51:23.344 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T11:51:23.344 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T11:51:23.344 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T11:51:23.344 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T11:51:23.345 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a[41341]: ts=2024-03-24T11:51:23.056Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T11:51:23.345 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 podman[43882]: 2024-03-24 11:51:23.066521459 +0000 UTC m=+0.110283759 container died 6e5961b3423f7c92c9fe9b2e61fbda60d4cafde60a7eef1a4874e744dd036130 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:23.778 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 podman[43882]: 2024-03-24 11:51:23.519601947 +0000 UTC m=+0.563364289 container cleanup 6e5961b3423f7c92c9fe9b2e61fbda60d4cafde60a7eef1a4874e744dd036130 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:23.778 INFO:journalctl@ceph.prometheus.a.smithi174.stdout:Mar 24 11:51:23 smithi174 bash[43882]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T11:51:23.930 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-24T11:51:23.965 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:51:23.966 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-24T11:51:23.966 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T11:51:24.155 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:51:25.395 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:51:25 smithi130 systemd[1]: Stopping Ceph alertmanager.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:51:25.682 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:51:25 smithi130 ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a[54503]: ts=2024-03-24T11:51:25.395Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:51:25.682 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:51:25 smithi130 podman[57574]: 2024-03-24 11:51:25.426013986 +0000 UTC m=+0.142966892 container died 363631a85202c110ce6fd6d4eb889331900e265ff4dd5e1270f645b903ae6052 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:26.026 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:51:25 smithi130 podman[57574]: 2024-03-24 11:51:25.867734988 +0000 UTC m=+0.584687936 container cleanup 363631a85202c110ce6fd6d4eb889331900e265ff4dd5e1270f645b903ae6052 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:26.026 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:51:25 smithi130 bash[57574]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T11:51:26.287 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:51:26 smithi130 podman[57593]: 2024-03-24 11:51:26.027084981 +0000 UTC m=+0.601955220 container remove 363631a85202c110ce6fd6d4eb889331900e265ff4dd5e1270f645b903ae6052 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:26.287 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:51:26 smithi130 podman[57593]: 2024-03-24 11:51:26.10263421 +0000 UTC m=+0.677504459 volume remove 545f5dedeecc4b079c1b054f2c0b89a1707c24a74e691dde38a17331d35186a0 2024-03-24T11:51:26.751 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:51:26 smithi130 systemd[1]: Stopping Ceph node-exporter.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:51:26.753 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:51:26 smithi130 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T11:51:26.753 INFO:journalctl@ceph.alertmanager.a.smithi130.stdout:Mar 24 11:51:26 smithi130 systemd[1]: Stopped Ceph alertmanager.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:51:27.037 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:51:26 smithi130 podman[57710]: 2024-03-24 11:51:26.751549715 +0000 UTC m=+0.093002097 container died 360cf4dfbc7612d128eeab808530f25599972b56b4649071295ebedead16dbc1 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:27.373 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:51:27 smithi130 podman[57710]: 2024-03-24 11:51:27.180265316 +0000 UTC m=+0.521717704 container cleanup 360cf4dfbc7612d128eeab808530f25599972b56b4649071295ebedead16dbc1 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:27.373 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:51:27 smithi130 bash[57710]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a 2024-03-24T11:51:27.738 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:51:27 smithi130 podman[57730]: 2024-03-24 11:51:27.373290331 +0000 UTC m=+0.626813850 container remove 360cf4dfbc7612d128eeab808530f25599972b56b4649071295ebedead16dbc1 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:51:27.738 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:51:27 smithi130 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:51:27.738 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:51:27 smithi130 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-24T11:51:27.738 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:51:27 smithi130 systemd[1]: Stopped Ceph node-exporter.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:51:27.738 INFO:journalctl@ceph.node-exporter.a.smithi130.stdout:Mar 24 11:51:27 smithi130 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 1.793s CPU time. 2024-03-24T11:51:28.222 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T11:51:28.413 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:51:29.343 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:51:29 smithi174 systemd[1]: Stopping Ceph grafana.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715... 2024-03-24T11:51:29.773 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:51:29 smithi174 podman[44219]: 2024-03-24 11:51:29.50387569 +0000 UTC m=+0.127077760 container died f9e18d02cf2d8ab5bed20c0b134e41c2d56435e2526ce6064e21122f70347d5f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, io.openshift.tags=base rhel8, name=ubi8, vcs-type=git, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, io.buildah.version=1.32.0, io.openshift.expose-services=, com.redhat.component=ubi8-container, distribution-scope=public, version=8.5, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-03-24T11:51:30.093 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:51:29 smithi174 podman[44219]: 2024-03-24 11:51:29.847686125 +0000 UTC m=+0.470888194 container cleanup f9e18d02cf2d8ab5bed20c0b134e41c2d56435e2526ce6064e21122f70347d5f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, vendor=Red Hat, Inc., description=Ceph Grafana Container, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, com.redhat.component=ubi8-container, io.openshift.expose-services=, name=ubi8) 2024-03-24T11:51:30.094 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:51:29 smithi174 bash[44219]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a 2024-03-24T11:51:30.408 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:51:30 smithi174 podman[44239]: 2024-03-24 11:51:30.123719271 +0000 UTC m=+0.622666842 container remove f9e18d02cf2d8ab5bed20c0b134e41c2d56435e2526ce6064e21122f70347d5f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715-grafana-a, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, name=ubi8, vendor=Red Hat, Inc., architecture=x86_64, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-03-24T11:51:30.408 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:51:30 smithi174 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:51:30.408 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:51:30 smithi174 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-24T11:51:30.408 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:51:30 smithi174 systemd[1]: Stopped Ceph grafana.a for e18cf2c8-e9d3-11ee-95cd-87774f69a715. 2024-03-24T11:51:30.408 INFO:journalctl@ceph.grafana.a.smithi174.stdout:Mar 24 11:51:30 smithi174 systemd[1]: ceph-e18cf2c8-e9d3-11ee-95cd-87774f69a715@grafana.a.service: Consumed 2.634s CPU time. 2024-03-24T11:51:30.888 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T11:51:31.078 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:51:32.138 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:51:32.174 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:51:32.208 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:51:32.236 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T11:51:32.238 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938/remote/smithi130/crash 2024-03-24T11:51:32.239 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:51:32.281 INFO:teuthology.orchestra.run.smithi130.stderr:tar: /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T11:51:32.281 INFO:teuthology.orchestra.run.smithi130.stderr:tar: Error is not recoverable: exiting now 2024-03-24T11:51:32.286 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938/remote/smithi174/crash 2024-03-24T11:51:32.287 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:51:32.333 INFO:teuthology.orchestra.run.smithi174.stderr:tar: /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T11:51:32.333 INFO:teuthology.orchestra.run.smithi174.stderr:tar: Error is not recoverable: exiting now 2024-03-24T11:51:32.335 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938/remote/smithi189/crash 2024-03-24T11:51:32.336 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:51:32.379 INFO:teuthology.orchestra.run.smithi189.stderr:tar: /var/lib/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T11:51:32.379 INFO:teuthology.orchestra.run.smithi189.stderr:tar: Error is not recoverable: exiting now 2024-03-24T11:51:32.380 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T11:51:32.380 DEBUG:teuthology.orchestra.run.smithi130:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:51:32.413 INFO:teuthology.orchestra.run.smithi130.stdout:2024-03-24T11:47:46.653387+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:51:32.413 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T11:51:32.413 DEBUG:teuthology.orchestra.run.smithi130:> sudo egrep '\[SEC\]' /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:51:32.483 DEBUG:teuthology.orchestra.run.smithi130:> sudo egrep '\[ERR\]' /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:51:32.554 DEBUG:teuthology.orchestra.run.smithi130:> sudo egrep '\[WRN\]' /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:51:32.623 INFO:teuthology.orchestra.run.smithi130.stdout:2024-03-24T11:47:46.653387+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:51:32.623 INFO:tasks.cephadm:Compressing logs... 2024-03-24T11:51:32.623 DEBUG:teuthology.orchestra.run.smithi130:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:51:32.665 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:51:32.668 DEBUG:teuthology.orchestra.run.smithi189:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:51:32.690 INFO:teuthology.orchestra.run.smithi130.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:51:32.690 INFO:teuthology.orchestra.run.smithi130.stderr:: No such file or directory 2024-03-24T11:51:32.690 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T11:51:32.690 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:51:32.690 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-24T11:51:32.694 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mon.a.log: /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T11:51:32.695 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T11:51:32.695 INFO:teuthology.orchestra.run.smithi130.stderr: 57.7% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-24T11:51:32.695 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:51:32.695 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:51:32.695 INFO:teuthology.orchestra.run.smithi130.stderr: 89.4% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:51:32.695 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:51:32.695 INFO:teuthology.orchestra.run.smithi130.stderr: 90.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:51:32.695 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log: 82.2% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:51:32.695 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-volume.log: 84.9% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:51:32.696 INFO:teuthology.orchestra.run.smithi130.stderr: 90.5% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:51:32.701 INFO:teuthology.orchestra.run.smithi189.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:51:32.701 INFO:teuthology.orchestra.run.smithi189.stderr:: No such file or directory 2024-03-24T11:51:32.701 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:51:32.701 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-24T11:51:32.701 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:51:32.701 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:51:32.701 INFO:teuthology.orchestra.run.smithi189.stderr: 90.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:51:32.701 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T11:51:32.702 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log: /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:51:32.702 INFO:teuthology.orchestra.run.smithi189.stderr: 80.4% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:51:32.702 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-osd.2.log: 89.6% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:51:32.702 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T11:51:32.702 INFO:teuthology.orchestra.run.smithi189.stderr: 84.5% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:51:32.702 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mon.c.log: 90.1% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-24T11:51:32.702 INFO:teuthology.orchestra.run.smithi189.stderr: 90.5% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:51:32.702 INFO:teuthology.orchestra.run.smithi174.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:: No such file or directory 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.audit.log: /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mgr.x.log: /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr: 80.4% 89.5% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log: -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-volume.log: 84.3% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr: 81.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mon.b.log: 90.5% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:51:32.703 INFO:teuthology.orchestra.run.smithi174.stderr: 90.2% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T11:51:32.749 INFO:teuthology.orchestra.run.smithi189.stderr: 92.2% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T11:51:32.755 INFO:teuthology.orchestra.run.smithi130.stderr: 89.0% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T11:51:32.761 INFO:teuthology.orchestra.run.smithi174.stderr: 92.3% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T11:51:32.861 INFO:teuthology.orchestra.run.smithi130.stderr: 91.4% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T11:51:32.938 INFO:teuthology.orchestra.run.smithi130.stderr: 94.3% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T11:51:32.940 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-24T11:51:32.940 INFO:teuthology.orchestra.run.smithi130.stderr:real 0m0.262s 2024-03-24T11:51:32.940 INFO:teuthology.orchestra.run.smithi130.stderr:user 0m0.486s 2024-03-24T11:51:32.940 INFO:teuthology.orchestra.run.smithi130.stderr:sys 0m0.035s 2024-03-24T11:51:32.950 INFO:teuthology.orchestra.run.smithi189.stderr: 94.2% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T11:51:32.952 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-24T11:51:32.952 INFO:teuthology.orchestra.run.smithi189.stderr:real 0m0.270s 2024-03-24T11:51:32.952 INFO:teuthology.orchestra.run.smithi189.stderr:user 0m0.318s 2024-03-24T11:51:32.952 INFO:teuthology.orchestra.run.smithi189.stderr:sys 0m0.032s 2024-03-24T11:51:33.010 INFO:teuthology.orchestra.run.smithi174.stderr: 94.3% -- replaced with /var/log/ceph/e18cf2c8-e9d3-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T11:51:33.012 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-03-24T11:51:33.012 INFO:teuthology.orchestra.run.smithi174.stderr:real 0m0.331s 2024-03-24T11:51:33.012 INFO:teuthology.orchestra.run.smithi174.stderr:user 0m0.386s 2024-03-24T11:51:33.012 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m0.037s 2024-03-24T11:51:33.012 INFO:tasks.cephadm:Archiving logs... 2024-03-24T11:51:33.012 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938/remote/smithi130/log 2024-03-24T11:51:33.013 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T11:51:33.129 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938/remote/smithi174/log 2024-03-24T11:51:33.130 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T11:51:33.214 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938/remote/smithi189/log 2024-03-24T11:51:33.215 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T11:51:33.291 INFO:tasks.cephadm:Removing cluster... 2024-03-24T11:51:33.291 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 --force 2024-03-24T11:51:33.468 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:51:33.586 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 --force 2024-03-24T11:51:33.768 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:51:33.880 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e18cf2c8-e9d3-11ee-95cd-87774f69a715 --force 2024-03-24T11:51:34.064 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: e18cf2c8-e9d3-11ee-95cd-87774f69a715 2024-03-24T11:51:34.178 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T11:51:34.179 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T11:51:34.197 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T11:51:34.211 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T11:51:34.226 INFO:tasks.cephadm:Teardown complete 2024-03-24T11:51:34.227 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-24T11:51:34.238 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi130:/dev/vg_nvme/lv_1... 2024-03-24T11:51:34.238 DEBUG:teuthology.orchestra.run.smithi130:> sudo nvme disconnect -n lv_1 2024-03-24T11:51:34.381 INFO:teuthology.orchestra.run.smithi130.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-24T11:51:34.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:34.383 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi130:/dev/vg_nvme/lv_2... 2024-03-24T11:51:34.383 DEBUG:teuthology.orchestra.run.smithi130:> sudo nvme disconnect -n lv_2 2024-03-24T11:51:34.521 INFO:teuthology.orchestra.run.smithi130.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-24T11:51:34.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:34.523 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi130:/dev/vg_nvme/lv_3... 2024-03-24T11:51:34.523 DEBUG:teuthology.orchestra.run.smithi130:> sudo nvme disconnect -n lv_3 2024-03-24T11:51:34.667 INFO:teuthology.orchestra.run.smithi130.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-24T11:51:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:34.669 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi130:/dev/vg_nvme/lv_4... 2024-03-24T11:51:34.669 DEBUG:teuthology.orchestra.run.smithi130:> sudo nvme disconnect -n lv_4 2024-03-24T11:51:34.815 INFO:teuthology.orchestra.run.smithi130.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-24T11:51:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:34.817 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-24T11:51:34.817 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/scratch_devs 2024-03-24T11:51:34.847 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi174:/dev/vg_nvme/lv_1... 2024-03-24T11:51:34.847 DEBUG:teuthology.orchestra.run.smithi174:> sudo nvme disconnect -n lv_1 2024-03-24T11:51:34.981 INFO:teuthology.orchestra.run.smithi174.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-24T11:51:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:34.983 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi174:/dev/vg_nvme/lv_2... 2024-03-24T11:51:34.983 DEBUG:teuthology.orchestra.run.smithi174:> sudo nvme disconnect -n lv_2 2024-03-24T11:51:35.123 INFO:teuthology.orchestra.run.smithi174.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-24T11:51:35.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:35.125 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi174:/dev/vg_nvme/lv_3... 2024-03-24T11:51:35.125 DEBUG:teuthology.orchestra.run.smithi174:> sudo nvme disconnect -n lv_3 2024-03-24T11:51:35.261 INFO:teuthology.orchestra.run.smithi174.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-24T11:51:35.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:35.263 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi174:/dev/vg_nvme/lv_4... 2024-03-24T11:51:35.263 DEBUG:teuthology.orchestra.run.smithi174:> sudo nvme disconnect -n lv_4 2024-03-24T11:51:35.399 INFO:teuthology.orchestra.run.smithi174.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-24T11:51:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:35.401 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-03-24T11:51:35.401 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/scratch_devs 2024-03-24T11:51:35.427 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_1... 2024-03-24T11:51:35.427 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_1 2024-03-24T11:51:35.575 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-24T11:51:35.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:35.577 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_2... 2024-03-24T11:51:35.577 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_2 2024-03-24T11:51:35.713 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-24T11:51:35.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:35.715 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_3... 2024-03-24T11:51:35.715 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_3 2024-03-24T11:51:35.865 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-24T11:51:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:35.867 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_4... 2024-03-24T11:51:35.867 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_4 2024-03-24T11:51:36.003 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-24T11:51:36.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:36.005 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:51:36.005 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/scratch_devs 2024-03-24T11:51:36.032 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T11:51:36.042 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T11:51:36.043 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:51:36.045 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:51:36.047 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:51:36.059 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:51:36.060 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:51:36.064 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:51:36.065 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-03-24T11:51:36.065 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +947us[ +947us] +/- 80ms 2024-03-24T11:51:36.065 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 +108us[ +114us] +/- 61ms 2024-03-24T11:51:36.065 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -829us[ -829us] +/- 64ms 2024-03-24T11:51:36.065 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:51:36.066 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:51:36.066 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-03-24T11:51:36.066 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +920us[ +928us] +/- 80ms 2024-03-24T11:51:36.066 INFO:teuthology.orchestra.run.smithi130.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 +83us[ +91us] +/- 61ms 2024-03-24T11:51:36.066 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 -845us[ -837us] +/- 64ms 2024-03-24T11:51:36.066 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:51:36.085 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:51:36.090 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:51:36.090 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-24T11:51:36.090 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +945us[ +956us] +/- 81ms 2024-03-24T11:51:36.090 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 +108us[ +119us] +/- 61ms 2024-03-24T11:51:36.090 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -817us[ -817us] +/- 64ms 2024-03-24T11:51:36.090 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:51:36.091 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T11:51:36.100 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T11:51:36.100 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T11:51:36.109 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:51:36.188 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:51:36.257 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:51:36.325 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:51:36.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:36.359 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:51:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:36.389 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:51:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:36.419 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T11:51:36.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T11:51:36.437 INFO:teuthology.task.internal:Duration was 694.046529 seconds 2024-03-24T11:51:36.437 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T11:51:36.445 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T11:51:36.446 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:51:36.448 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:51:36.450 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:51:36.491 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:51:36.492 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:51:36.502 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:51:36.878 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T11:51:36.879 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2024-03-24T11:51:36.879 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T11:51:36.899 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-03-24T11:51:36.900 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T11:51:36.945 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-03-24T11:51:36.946 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T11:51:36.967 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T11:51:36.967 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T11:51:36.970 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T11:51:36.988 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T11:51:37.036 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T11:51:37.036 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T11:51:37.040 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T11:51:37.056 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T11:51:37.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T11:51:37.239 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T11:51:37.239 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:51:37.265 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:51:37.292 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:51:37.319 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T11:51:37.329 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:51:37.331 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:51:37.335 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:51:37.354 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2024-03-24T11:51:37.358 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-03-24T11:51:37.384 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-03-24T11:51:37.411 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:51:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:37.456 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:51:37.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:37.470 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:51:37.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:51:37.484 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T11:51:37.494 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T11:51:37.494 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938/remote/smithi130 2024-03-24T11:51:37.495 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T11:51:37.533 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938/remote/smithi174 2024-03-24T11:51:37.534 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T11:51:37.569 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938/remote/smithi189 2024-03-24T11:51:37.569 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T11:51:37.607 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T11:51:37.607 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T11:51:37.610 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T11:51:37.612 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T11:51:37.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T11:51:37.665 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T11:51:37.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T11:51:37.675 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T11:51:37.675 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T11:51:37.677 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T11:51:37.679 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T11:51:37.689 INFO:teuthology.orchestra.run.smithi130.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:51 /home/ubuntu/cephtest 2024-03-24T11:51:37.690 INFO:teuthology.orchestra.run.smithi174.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:51 /home/ubuntu/cephtest 2024-03-24T11:51:37.708 INFO:teuthology.orchestra.run.smithi189.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:51 /home/ubuntu/cephtest 2024-03-24T11:51:37.709 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T11:51:37.720 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T11:51:37.741 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T11:51:37.769 DEBUG:teuthology.nuke:shortname: smithi130 2024-03-24T11:51:37.769 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:51:37.793 DEBUG:teuthology.nuke:shortname: smithi174 2024-03-24T11:51:37.793 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:51:37.798 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:29:31.896685', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBnNasDdYlxjP264AtAYghJBzdsnLAX8t+0AKvqGcs9IJBfqj2ipsYsTC1yzPnbJWWY1XZStiXWRA2nqrQnWClg='} 2024-03-24T11:51:37.816 DEBUG:teuthology.nuke:shortname: smithi189 2024-03-24T11:51:37.816 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:51:37.821 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:29:31.897924', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBImy5hzhgK6dxJSSTzKRdpyWRTgeFmIeqf/BpTsHkd2ZIF/ariilZrnGUcxh32fQLSyW2vs2antgV7QBRpBvXYQ='} 2024-03-24T11:51:37.851 INFO:teuthology.orchestra.console.smithi130:Power off 2024-03-24T11:51:37.851 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:51:37.860 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617938', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:29:31.895160', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdWjCr8lwue03NKbJADyVIQ0h7TnSC0QizrEu49mmF4Le2cv/0dpMzkbE686tHQp+4FiWWJO9GrNsStsk/evGk='} 2024-03-24T11:51:37.875 DEBUG:teuthology.orchestra.console.smithi130:power off output: Chassis Power Control: Down/Off 2024-03-24T11:51:37.876 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:51:37.898 INFO:teuthology.orchestra.console.smithi174:Power off 2024-03-24T11:51:37.898 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:51:37.908 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-03-24T11:51:37.923 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off 2024-03-24T11:51:37.923 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:51:37.938 INFO:teuthology.orchestra.console.smithi189:Power off 2024-03-24T11:51:37.938 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:51:37.948 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-03-24T11:51:37.960 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-03-24T11:51:37.960 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:51:37.983 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-24T11:51:41.910 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:51:41.949 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:51:41.984 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:51:42.020 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-03-24T11:51:42.059 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-03-24T11:51:42.094 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-24T11:51:46.024 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:51:46.060 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:51:46.098 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:51:46.133 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-03-24T11:51:46.167 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off 2024-03-24T11:51:46.208 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-24T11:51:46.234 INFO:teuthology.orchestra.console.smithi130:Power off completed 2024-03-24T11:51:46.268 INFO:teuthology.orchestra.console.smithi174:Power off completed 2024-03-24T11:51:50.210 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:51:50.319 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-03-24T11:51:50.419 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-03-24T11:51:50.523 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} duration: 694.0465288162231 failure_reason: '"2024-03-24T11:47:46.653387+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-24T11:51:50.524 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:51:50.617 INFO:teuthology.run:FAIL