2024-03-26T01:56:40.722 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T01:56:40.722 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_teuthology@teuthology --name teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} --archive /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944 -- /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944/orig.config.yaml 2024-03-26T01:56:40.750 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T01:56:40.835 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7621944' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 1822 sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a sleep_before_teardown: 0 subset: 53/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de targets: smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGpKHj+k2YawkL2g51IJ5F6/gsdHapiJ3mfoYB4MyFTcY6/WvDvTaDXCL7p9KqWiPUGpiN0syMKDa/DvuqPdiZA= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGhgBLD76fFQ2m3Ju/nyHCweGwAPTaXT9JJBD5DN1VDc6t+HrnUijm55twL5z9hf8lKkfbkymoYZiyr1MCQAiPM= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIvncD+o5SYzAB3IgYtAx0P0I3yCWKaEQaJrn4aRIx97SFsHDXIjhiZXWGZsRoZu8GBRl4qLZhwviRop51kA70M= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_21:08:11 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T01:56:40.836 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa; will attempt to use it 2024-03-26T01:56:40.836 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa/tasks 2024-03-26T01:56:40.836 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T01:56:40.838 INFO:teuthology.task.internal:Checking packages... 2024-03-26T01:56:40.864 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e70124b46c2a671efcaae4690cd9ea72670710a' 2024-03-26T01:56:40.864 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T01:56:40.864 INFO:teuthology.packaging:ref: None 2024-03-26T01:56:40.864 INFO:teuthology.packaging:tag: None 2024-03-26T01:56:40.864 INFO:teuthology.packaging:branch: squid 2024-03-26T01:56:40.864 INFO:teuthology.packaging:sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:56:40.864 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-26T01:56:41.032 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1863.g9f04c91b 2024-03-26T01:56:41.034 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T01:56:41.039 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T01:56:41.039 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T01:56:41.095 INFO:teuthology.task.internal:Saving configuration 2024-03-26T01:56:41.106 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T01:56:41.120 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T01:56:41.144 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:51:14.254090', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGpKHj+k2YawkL2g51IJ5F6/gsdHapiJ3mfoYB4MyFTcY6/WvDvTaDXCL7p9KqWiPUGpiN0syMKDa/DvuqPdiZA='} 2024-03-26T01:56:41.168 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:51:14.251213', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGhgBLD76fFQ2m3Ju/nyHCweGwAPTaXT9JJBD5DN1VDc6t+HrnUijm55twL5z9hf8lKkfbkymoYZiyr1MCQAiPM='} 2024-03-26T01:56:41.193 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:51:14.252740', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIvncD+o5SYzAB3IgYtAx0P0I3yCWKaEQaJrn4aRIx97SFsHDXIjhiZXWGZsRoZu8GBRl4qLZhwviRop51kA70M='} 2024-03-26T01:56:41.193 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T01:56:41.199 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T01:56:41.199 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-26T01:56:41.199 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-26T01:56:41.199 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T01:56:41.325 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f45d405af70>, signals=[15]) 2024-03-26T01:56:41.326 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T01:56:41.332 INFO:teuthology.task.internal:Opening connections... 2024-03-26T01:56:41.332 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-03-26T01:56:41.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:56:41.441 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-03-26T01:56:41.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:56:41.531 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-03-26T01:56:41.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:56:41.621 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T01:56:41.627 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-03-26T01:56:41.645 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-03-26T01:56:41.646 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-03-26T01:56:41.703 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-03-26T01:56:41.703 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2024-03-26T01:56:41.703 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-03-26T01:56:41.703 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-03-26T01:56:41.703 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2024-03-26T01:56:41.703 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2024-03-26T01:56:41.703 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T01:56:41.703 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-03-26T01:56:41.703 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2024-03-26T01:56:41.704 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T01:56:41.704 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-03-26T01:56:41.704 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T01:56:41.704 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T01:56:41.704 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T01:56:41.704 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-03-26T01:56:41.730 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-03-26T01:56:41.748 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-03-26T01:56:41.748 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-03-26T01:56:41.805 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-03-26T01:56:41.805 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="9" 2024-03-26T01:56:41.805 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-03-26T01:56:41.805 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-03-26T01:56:41.805 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="9" 2024-03-26T01:56:41.805 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el9" 2024-03-26T01:56:41.805 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T01:56:41.805 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-03-26T01:56:41.805 INFO:teuthology.orchestra.run.smithi165.stdout:LOGO="fedora-logo-icon" 2024-03-26T01:56:41.805 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T01:56:41.805 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-03-26T01:56:41.806 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T01:56:41.806 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T01:56:41.806 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T01:56:41.806 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-03-26T01:56:41.830 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-03-26T01:56:41.847 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-03-26T01:56:41.848 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-03-26T01:56:41.905 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-03-26T01:56:41.905 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-03-26T01:56:41.905 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-03-26T01:56:41.905 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-03-26T01:56:41.905 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-03-26T01:56:41.905 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-03-26T01:56:41.905 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T01:56:41.905 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-03-26T01:56:41.905 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-03-26T01:56:41.906 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T01:56:41.906 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-03-26T01:56:41.906 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T01:56:41.906 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T01:56:41.906 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T01:56:41.906 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-03-26T01:56:41.930 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T01:56:41.940 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T01:56:41.964 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T01:56:41.964 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-03-26T01:56:41.970 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-03-26T01:56:41.974 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-03-26T01:56:41.990 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T01:56:42.013 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T01:56:42.014 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-03-26T01:56:42.028 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-03-26T01:56:42.030 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-03-26T01:56:42.120 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T01:56:42.186 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T01:56:42.220 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T01:56:42.220 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T01:56:42.221 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T01:56:42.221 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T01:56:42.222 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T01:56:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:56:42.238 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-03-26T01:56:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:56:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:56:42.243 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-03-26T01:56:42.244 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-03-26T01:56:42.296 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-361.el9.x86_64 2024-03-26T01:56:42.296 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.14.0-361.el9.x86_64 2024-03-26T01:56:42.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum install -y kernel 2024-03-26T01:56:42.301 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-361.el9.x86_64 2024-03-26T01:56:42.301 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.14.0-361.el9.x86_64 2024-03-26T01:56:42.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2024-03-26T01:56:42.302 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-361.el9.x86_64 2024-03-26T01:56:42.302 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-361.el9.x86_64 2024-03-26T01:56:42.302 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2024-03-26T01:56:44.474 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-26T01:56:44.476 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T01:56:44.552 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-26T01:56:45.935 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-26T01:56:45.958 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-26T01:56:45.978 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-26T01:56:47.407 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 7.2 MB/s | 8.0 MB 00:01 2024-03-26T01:56:47.911 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.0 MB 00:01 2024-03-26T01:56:48.069 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-03-26T01:56:49.058 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 33 MB/s | 19 MB 00:00 2024-03-26T01:56:49.719 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-26T01:56:52.678 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 5.4 MB/s | 19 MB 00:03 2024-03-26T01:56:53.786 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-26T01:56:54.033 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-26T01:56:55.253 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-26T01:56:56.342 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-26T01:56:57.062 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.1 MB 00:00 2024-03-26T01:56:57.568 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-26T01:56:58.554 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-26T01:56:59.049 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 5.6 MB/s | 21 MB 00:03 2024-03-26T01:57:00.575 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-26T01:57:02.517 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-26T01:57:03.996 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-26T01:57:04.683 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T01:57:04.683 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T01:57:04.779 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-03-26T01:57:04.783 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-26T01:57:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-03-26T01:57:04.783 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-26T01:57:04.784 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-03-26T01:57:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T01:57:04.784 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-03-26T01:57:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T01:57:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T01:57:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T01:57:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-26T01:57:04.784 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-03-26T01:57:04.784 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-26T01:57:04.784 INFO:teuthology.orchestra.run.smithi165.stdout:Install 4 Packages 2024-03-26T01:57:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-26T01:57:04.785 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 76 M 2024-03-26T01:57:04.785 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 123 M 2024-03-26T01:57:04.785 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-03-26T01:57:05.010 INFO:teuthology.orchestra.run.smithi165.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 620 kB/s | 66 kB 00:00 2024-03-26T01:57:05.394 INFO:teuthology.orchestra.run.smithi165.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 30 MB/s | 15 MB 00:00 2024-03-26T01:57:05.681 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-26T01:57:05.802 INFO:teuthology.orchestra.run.smithi165.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 37 MB/s | 33 MB 00:00 2024-03-26T01:57:06.053 INFO:teuthology.orchestra.run.smithi165.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 26 MB/s | 27 MB 00:01 2024-03-26T01:57:06.053 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:57:06.053 INFO:teuthology.orchestra.run.smithi165.stdout:Total 60 MB/s | 76 MB 00:01 2024-03-26T01:57:06.145 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T01:57:06.145 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T01:57:06.229 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-26T01:57:06.233 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-26T01:57:06.233 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-03-26T01:57:06.233 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-26T01:57:06.233 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 76 M 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 123 M 2024-03-26T01:57:06.234 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-26T01:57:06.722 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-03-26T01:57:06.805 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 219 kB/s | 66 kB 00:00 2024-03-26T01:57:07.103 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-03-26T01:57:07.103 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-03-26T01:57:07.456 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-03-26T01:57:07.889 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 24 MB/s | 33 MB 00:01 2024-03-26T01:57:07.904 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T01:57:07.904 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T01:57:07.996 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-26T01:57:08.004 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-26T01:57:08.004 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-03-26T01:57:08.004 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-26T01:57:08.004 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-26T01:57:08.004 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T01:57:08.004 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-26T01:57:08.004 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T01:57:08.004 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T01:57:08.004 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T01:57:08.005 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T01:57:08.005 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-26T01:57:08.005 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-26T01:57:08.005 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-03-26T01:57:08.005 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T01:57:08.005 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 76 M 2024-03-26T01:57:08.005 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 123 M 2024-03-26T01:57:08.006 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-26T01:57:08.124 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-03-26T01:57:08.124 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-03-26T01:57:08.142 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:57:08.142 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:57:08.143 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:57:08.284 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:57:08.298 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:57:08.320 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:57:08.482 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 16 MB/s | 27 MB 00:01 2024-03-26T01:57:08.482 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:57:08.483 INFO:teuthology.orchestra.run.smithi191.stdout:Total 34 MB/s | 76 MB 00:02 2024-03-26T01:57:08.632 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 113 kB/s | 66 kB 00:00 2024-03-26T01:57:08.665 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:57:08.698 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:57:09.150 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-26T01:57:09.516 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-26T01:57:09.516 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-26T01:57:10.502 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-26T01:57:10.503 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-26T01:57:10.661 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-03-26T01:57:11.338 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:57:11.426 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:57:12.965 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-26T01:57:13.605 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:57:13.699 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:57:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:57:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:57:15.849 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 4.3 MB/s | 33 MB 00:07 2024-03-26T01:57:16.290 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 3.6 MB/s | 27 MB 00:07 2024-03-26T01:57:16.458 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:57:16.595 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:57:19.058 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:57:19.278 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:20.815 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): kernel-core-5.14.0-432.el9.x86_64.rpm 1.2 MB/s | 15 MB 00:12 2024-03-26T01:57:20.818 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:57:20.819 INFO:teuthology.orchestra.run.smithi052.stdout:Total 5.9 MB/s | 76 MB 00:12 2024-03-26T01:57:21.321 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:57:21.445 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:21.510 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-26T01:57:21.944 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-26T01:57:21.944 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-26T01:57:22.977 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-26T01:57:22.977 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-26T01:57:24.023 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:25.589 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-26T01:57:26.189 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:26.225 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:57:26.316 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:57:29.300 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:57:29.436 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:57:34.510 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:57:34.553 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:39.659 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:53.469 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:53.469 INFO:teuthology.orchestra.run.smithi165.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T01:57:53.469 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-26T01:57:53.513 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:53.864 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:53.864 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:57:53.864 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:57:53.864 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:57:54.233 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:54.233 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-26T01:57:54.233 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-03-26T01:57:54.233 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T01:57:54.233 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T01:57:54.233 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T01:57:54.233 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T01:57:54.234 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-26T01:57:54.234 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-03-26T01:57:54.560 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:57:54.662 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:57:54.662 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:57:54.663 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T01:57:54.663 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T01:57:54.663 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T01:57:54.663 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T01:57:54.663 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi165.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T01:57:54.663 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:57:54.743 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:57:54.743 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:57:54.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo rpm -qi grub2-tools 2024-03-26T01:57:54.784 INFO:teuthology.orchestra.run.smithi165.stdout:Name : grub2-tools 2024-03-26T01:57:54.784 INFO:teuthology.orchestra.run.smithi165.stdout:Epoch : 1 2024-03-26T01:57:54.784 INFO:teuthology.orchestra.run.smithi165.stdout:Version : 2.06 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Release : 68.el9 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: x86_64 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Group : Unspecified 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Size : 8264254 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:License : GPLv3+ 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Packager : builder@centos.org 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Vendor : CentOS 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Summary : Support tools for GRUB. 2024-03-26T01:57:54.785 INFO:teuthology.orchestra.run.smithi165.stdout:Description : 2024-03-26T01:57:54.786 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-26T01:57:54.786 INFO:teuthology.orchestra.run.smithi165.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T01:57:54.786 INFO:teuthology.orchestra.run.smithi165.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T01:57:54.786 INFO:teuthology.orchestra.run.smithi165.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T01:57:54.786 INFO:teuthology.orchestra.run.smithi165.stdout:hardware devices. 2024-03-26T01:57:54.786 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-26T01:57:54.786 INFO:teuthology.orchestra.run.smithi165.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T01:57:54.787 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T01:57:54.787 INFO:teuthology.task.kernel:Updating grub on smithi165 to boot 5.14.0-432.el9.x86_64 2024-03-26T01:57:54.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T01:57:55.453 INFO:teuthology.orchestra.run.smithi165.stderr:Generating grub configuration file ... 2024-03-26T01:57:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:55.683 INFO:teuthology.orchestra.run.smithi191.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T01:57:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T01:57:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:56.122 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:56.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:57:56.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:57:56.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:57:56.546 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:57:56.546 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T01:57:56.546 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-26T01:57:56.546 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T01:57:56.546 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T01:57:56.546 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T01:57:56.546 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T01:57:56.546 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T01:57:56.546 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-26T01:57:56.882 INFO:teuthology.orchestra.run.smithi165.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T01:57:56.906 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:57:56.915 INFO:teuthology.orchestra.run.smithi165.stderr:done 2024-03-26T01:57:56.917 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-03-26T01:57:56.972 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.WkRsbbxR8G 2024-03-26T01:57:56.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WkRsbbxR8G 2024-03-26T01:57:57.001 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:57:57.002 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:57:57.002 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T01:57:57.002 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T01:57:57.002 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T01:57:57.002 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-26T01:57:57.002 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T01:57:57.002 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:57:57.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.WkRsbbxR8G 2024-03-26T01:57:57.082 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:57:57.083 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:57:57.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2024-03-26T01:57:57.116 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2024-03-26T01:57:57.116 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.06 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 68.el9 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Group : Unspecified 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 8264254 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : builder@centos.org 2024-03-26T01:57:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2024-03-26T01:57:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T01:57:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2024-03-26T01:57:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2024-03-26T01:57:57.118 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T01:57:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T01:57:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T01:57:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T01:57:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2024-03-26T01:57:57.118 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T01:57:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T01:57:57.119 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T01:57:57.119 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 5.14.0-432.el9.x86_64 2024-03-26T01:57:57.119 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T01:57:57.179 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.WkRsbbxR8G is 6KB 2024-03-26T01:57:57.228 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.WkRsbbxR8G 2024-03-26T01:57:57.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T01:57:57.317 INFO:teuthology.orchestra.run.smithi165.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T01:57:57.317 INFO:teuthology.orchestra.run.smithi165.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T01:57:57.317 INFO:teuthology.orchestra.run.smithi165.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T01:57:57.318 INFO:teuthology.orchestra.run.smithi165.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T01:57:57.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T01:57:57.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo shutdown -r now 2024-03-26T01:57:57.754 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2024-03-26T01:57:59.125 INFO:teuthology.orchestra.run.smithi191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T01:57:59.156 INFO:teuthology.orchestra.run.smithi191.stderr:done 2024-03-26T01:57:59.159 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-03-26T01:57:59.174 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.ZRIUbicFDR 2024-03-26T01:57:59.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZRIUbicFDR 2024-03-26T01:57:59.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.ZRIUbicFDR 2024-03-26T01:57:59.381 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.ZRIUbicFDR is 6KB 2024-03-26T01:57:59.429 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.ZRIUbicFDR 2024-03-26T01:57:59.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T01:57:59.517 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T01:57:59.517 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T01:57:59.517 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T01:57:59.517 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T01:57:59.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T01:57:59.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2024-03-26T01:58:10.400 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:58:10.400 INFO:teuthology.orchestra.run.smithi052.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T01:58:10.400 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T01:58:10.448 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:58:10.956 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:58:10.956 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:58:10.956 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:58:10.956 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:58:11.383 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:58:11.383 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T01:58:11.383 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-26T01:58:11.383 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T01:58:11.383 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T01:58:11.383 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T01:58:11.383 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T01:58:11.383 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T01:58:11.384 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-26T01:58:11.722 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:58:11.829 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:58:11.829 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:58:11.829 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T01:58:11.829 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T01:58:11.830 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T01:58:11.830 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T01:58:11.830 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T01:58:11.830 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:58:11.912 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:58:11.913 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:58:11.913 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2024-03-26T01:58:11.950 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2024-03-26T01:58:11.950 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2024-03-26T01:58:11.950 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2024-03-26T01:58:11.950 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 68.el9 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8264254 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2024-03-26T01:58:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2024-03-26T01:58:11.952 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T01:58:11.952 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T01:58:11.952 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T01:58:11.952 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T01:58:11.952 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2024-03-26T01:58:11.952 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T01:58:11.952 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T01:58:11.953 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T01:58:11.953 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 5.14.0-432.el9.x86_64 2024-03-26T01:58:11.953 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T01:58:12.628 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2024-03-26T01:58:14.160 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T01:58:14.193 INFO:teuthology.orchestra.run.smithi052.stderr:done 2024-03-26T01:58:14.195 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-03-26T01:58:14.212 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.3V7Qpo1ArW 2024-03-26T01:58:14.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3V7Qpo1ArW 2024-03-26T01:58:14.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.3V7Qpo1ArW 2024-03-26T01:58:14.435 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.3V7Qpo1ArW is 6KB 2024-03-26T01:58:14.485 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.3V7Qpo1ArW 2024-03-26T01:58:14.500 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T01:58:14.569 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T01:58:14.569 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T01:58:14.569 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T01:58:14.569 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T01:58:14.570 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T01:58:14.727 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2024-03-26T01:58:27.475 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T01:58:27.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-26T01:58:27.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:58:29.682 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-26T01:58:29.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-26T01:58:29.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:58:44.742 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T01:58:44.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-26T01:58:44.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:58:46.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-03-26T01:58:55.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-26T01:58:55.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:58:58.114 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-03-26T01:59:03.202 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-03-26T01:59:10.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-26T01:59:10.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:59:12.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-26T01:59:12.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:59:29.703 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T01:59:38.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-26T01:59:38.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:59:40.187 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-26T01:59:40.730 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-26T01:59:40.731 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-432.el9.x86_64"... 2024-03-26T01:59:40.731 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-03-26T01:59:40.745 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-432.el9.x86_64 2024-03-26T01:59:40.745 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T01:59:40.745 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T01:59:40.745 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T01:59:41.745 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-26T01:59:41.746 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T01:59:41.817 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2024-03-26T01:59:41.855 DEBUG:teuthology.parallel:result is None 2024-03-26T02:00:10.122 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T02:00:12.209 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T02:00:24.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-26T02:00:24.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:00:24.536 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-03-26T02:00:24.909 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-26T02:00:24.909 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T02:00:24.910 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-03-26T02:00:24.964 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-432.el9.x86_64 2024-03-26T02:00:24.965 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T02:00:24.965 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T02:00:24.965 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T02:00:25.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-26T02:00:25.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:00:25.482 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-03-26T02:00:25.906 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-26T02:00:25.907 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T02:00:25.907 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-03-26T02:00:25.922 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-432.el9.x86_64 2024-03-26T02:00:25.923 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T02:00:25.923 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T02:00:25.923 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T02:00:25.965 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T02:00:25.966 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T02:00:26.038 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2024-03-26T02:00:26.092 DEBUG:teuthology.parallel:result is None 2024-03-26T02:00:26.924 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T02:00:26.924 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T02:00:27.007 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-03-26T02:00:27.040 DEBUG:teuthology.parallel:result is None 2024-03-26T02:00:27.040 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T02:00:27.048 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T02:00:27.048 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T02:00:27.051 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T02:00:27.053 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T02:00:27.073 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T02:00:27.090 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T02:00:27.133 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T02:00:27.133 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T02:00:27.136 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T02:00:27.138 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T02:00:27.188 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T02:00:27.226 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T02:00:27.226 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T02:00:27.229 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T02:00:27.232 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T02:00:27.263 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:00:27.274 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:00:27.277 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:00:27.277 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:00:27.288 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:00:27.292 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:00:27.294 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T02:00:27.301 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T02:00:27.301 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T02:00:27.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T02:00:27.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T02:00:27.353 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T02:00:27.365 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T02:00:27.365 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T02:00:27.392 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T02:00:27.395 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T02:00:27.410 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:00:27.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:00:27.539 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:00:27.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:00:27.664 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:00:27.664 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T02:00:27.731 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:00:27.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:00:27.833 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:00:27.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:00:27.960 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-26T02:00:27.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T02:00:28.028 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:00:28.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:00:28.137 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:00:28.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:00:28.225 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-26T02:00:28.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T02:00:28.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-03-26T02:00:28.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-03-26T02:00:28.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-03-26T02:00:28.339 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:00:28.341 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:00:28.394 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:00:28.745 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T02:00:28.753 INFO:teuthology.task.internal:Starting timer... 2024-03-26T02:00:28.753 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T02:00:28.804 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T02:00:28.852 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T02:00:28.853 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-03-26T02:00:28.906 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-03-26T02:00:28.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-03-26T02:00:28.978 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-03-26T02:00:28.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-03-26T02:00:29.057 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-03-26T02:00:29.059 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T02:00:29.059 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-03-26T02:00:29.081 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-03-26T02:00:29.081 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-03-26T02:00:29.104 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-03-26T02:00:29.104 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-03-26T02:00:29.125 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-03-26T02:00:29.125 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi052.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi165.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-03-26T02:00:29.125 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:00:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:29.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:00:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:29.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:00:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:00:29.214 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T02:00:29.215 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-03-26T02:00:29.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-03-26T02:00:29.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-03-26T02:00:29.325 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T02:00:29.336 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-26T02:00:29.341 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-26T02:00:29.560 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T02:00:29.582 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T02:00:29.583 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi165.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T02:05:02.078 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-03-26T02:05:02.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-26T02:05:02.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:05:02.171 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-03-26T02:05:02.249 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-26T02:05:02.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-26T02:05:02.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:05:02.338 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-03-26T02:05:02.418 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-26T02:05:02.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-26T02:05:02.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:05:02.511 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-26T02:05:02.598 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-26T02:05:02.598 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T02:05:02.609 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T02:05:02.609 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T02:05:02.610 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:05:02.612 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T02:05:02.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:05:02.614 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T02:05:02.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:05:02.643 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T02:05:02.648 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T02:05:02.660 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T02:05:02.666 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T02:05:02.671 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T02:05:02.688 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T02:05:02.690 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-03-26T02:05:02.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-03-26T02:05:02.704 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-03-26T02:05:02.711 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-03-26T02:05:02.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-03-26T02:05:02.721 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T02:05:02.729 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T02:05:02.732 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-03-26T02:05:02.737 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T02:05:02.745 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T02:05:02.749 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T02:05:02.766 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T02:05:02.790 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:05:02.794 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:05:02.794 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-03-26T02:05:02.795 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.795 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.795 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.795 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.796 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:05:02.799 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:05:02.799 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-03-26T02:05:02.799 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.800 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.800 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.800 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.822 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:05:02.826 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:05:02.826 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-26T02:05:02.826 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.826 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.826 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.826 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:05:02.827 INFO:teuthology.run_tasks:Running task pexec... 2024-03-26T02:05:02.836 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-26T02:05:02.836 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi052.front.sepia.ceph.com 2024-03-26T02:05:02.836 DEBUG:teuthology.orchestra.run.smithi052:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T02:05:02.837 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi165.front.sepia.ceph.com 2024-03-26T02:05:02.837 DEBUG:teuthology.orchestra.run.smithi165:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T02:05:02.837 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi191.front.sepia.ceph.com 2024-03-26T02:05:02.837 DEBUG:teuthology.orchestra.run.smithi191:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T02:05:03.420 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:04 ago on Tue 26 Mar 2024 02:03:59 AM UTC. 2024-03-26T02:05:03.427 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:02 ago on Tue 26 Mar 2024 02:04:01 AM UTC. 2024-03-26T02:05:03.449 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:04 ago on Tue 26 Mar 2024 02:03:59 AM UTC. 2024-03-26T02:05:03.567 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-03-26T02:05:03.569 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-26T02:05:03.569 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-03-26T02:05:03.569 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-26T02:05:03.569 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-03-26T02:05:03.569 INFO:teuthology.orchestra.run.smithi165.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T02:05:03.569 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-26T02:05:03.569 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-03-26T02:05:03.569 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-26T02:05:03.570 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-03-26T02:05:03.570 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-26T02:05:03.570 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 3.1 M 2024-03-26T02:05:03.570 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 9.8 M 2024-03-26T02:05:03.570 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-03-26T02:05:03.579 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-26T02:05:03.580 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-26T02:05:03.580 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-03-26T02:05:03.580 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-26T02:05:03.581 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-26T02:05:03.581 INFO:teuthology.orchestra.run.smithi052.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T02:05:03.581 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:05:03.581 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-26T02:05:03.581 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-26T02:05:03.581 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-26T02:05:03.581 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:05:03.581 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.1 M 2024-03-26T02:05:03.582 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 9.8 M 2024-03-26T02:05:03.582 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-26T02:05:03.595 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-26T02:05:03.613 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-26T02:05:03.613 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-03-26T02:05:03.613 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-26T02:05:03.613 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-26T02:05:03.613 INFO:teuthology.orchestra.run.smithi191.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T02:05:03.613 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T02:05:03.614 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-26T02:05:03.614 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-26T02:05:03.614 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-26T02:05:03.614 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T02:05:03.614 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 3.1 M 2024-03-26T02:05:03.614 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 9.8 M 2024-03-26T02:05:03.614 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-26T02:05:04.121 INFO:teuthology.orchestra.run.smithi165.stdout:runc-1.1.12-2.el9.x86_64.rpm 13 MB/s | 3.1 MB 00:00 2024-03-26T02:05:04.122 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:05:04.122 INFO:teuthology.orchestra.run.smithi165.stdout:Total 5.7 MB/s | 3.1 MB 00:00 2024-03-26T02:05:04.158 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-03-26T02:05:04.164 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-03-26T02:05:04.164 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-03-26T02:05:04.244 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-03-26T02:05:04.244 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-03-26T02:05:04.462 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-03-26T02:05:04.485 INFO:teuthology.orchestra.run.smithi191.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.4 MB/s | 3.1 MB 00:00 2024-03-26T02:05:04.486 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:05:04.486 INFO:teuthology.orchestra.run.smithi191.stdout:Total 3.6 MB/s | 3.1 MB 00:00 2024-03-26T02:05:04.522 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-26T02:05:04.528 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-26T02:05:04.528 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-26T02:05:04.557 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T02:05:04.574 INFO:teuthology.orchestra.run.smithi052.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.3 MB/s | 3.1 MB 00:00 2024-03-26T02:05:04.574 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:05:04.574 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.1 MB/s | 3.1 MB 00:00 2024-03-26T02:05:04.608 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-26T02:05:04.609 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-26T02:05:04.611 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-26T02:05:04.618 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-26T02:05:04.618 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-26T02:05:04.700 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-26T02:05:04.701 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-26T02:05:04.841 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-26T02:05:04.908 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-26T02:05:04.931 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T02:05:04.996 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T02:05:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T02:05:05.354 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T02:05:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-26T02:05:05.355 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-03-26T02:05:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T02:05:05.355 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-26T02:05:05.355 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-03-26T02:05:05.439 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T02:05:05.611 DEBUG:teuthology.parallel:result is None 2024-03-26T02:05:05.659 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T02:05:05.733 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T02:05:05.733 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T02:05:05.733 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-26T02:05:05.733 INFO:teuthology.orchestra.run.smithi191.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T02:05:05.733 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T02:05:05.733 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-26T02:05:05.981 DEBUG:teuthology.parallel:result is None 2024-03-26T02:05:05.984 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T02:05:05.984 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:05:05.984 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-26T02:05:05.984 INFO:teuthology.orchestra.run.smithi052.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T02:05:05.984 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:05:05.984 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-26T02:05:06.221 DEBUG:teuthology.parallel:result is None 2024-03-26T02:05:06.221 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-26T02:05:06.231 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-26T02:05:06.231 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:05:06.231 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:05:06.246 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T02:05:06.247 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2024-03-26T02:05:06.301 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T02:05:06.301 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:06.301 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T02:05:06.302 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:06.302 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:06.302 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-26 02:05:01.056200415 +0000 2024-03-26T02:05:06.302 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-26 02:04:26.670640260 +0000 2024-03-26T02:05:06.302 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-26 02:04:26.670640260 +0000 2024-03-26T02:05:06.302 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-26 02:04:26.670640260 +0000 2024-03-26T02:05:06.302 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T02:05:06.366 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-26T02:05:06.366 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-26T02:05:06.366 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.209e-05 s, 5.6 MB/s 2024-03-26T02:05:06.367 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T02:05:06.423 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2024-03-26T02:05:06.478 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T02:05:06.478 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:06.478 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T02:05:06.478 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:06.478 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:06.478 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-26 02:05:01.056200415 +0000 2024-03-26T02:05:06.478 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-26 02:04:27.005635975 +0000 2024-03-26T02:05:06.478 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-26 02:04:27.005635975 +0000 2024-03-26T02:05:06.478 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-26 02:04:27.005635975 +0000 2024-03-26T02:05:06.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T02:05:06.543 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-26T02:05:06.543 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-26T02:05:06.543 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000101587 s, 5.0 MB/s 2024-03-26T02:05:06.544 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T02:05:06.599 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2024-03-26T02:05:06.653 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T02:05:06.653 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:06.654 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-26T02:05:06.654 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:06.654 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:06.654 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-26 02:05:01.056200415 +0000 2024-03-26T02:05:06.654 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-26 02:04:27.330631818 +0000 2024-03-26T02:05:06.654 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-26 02:04:27.330631818 +0000 2024-03-26T02:05:06.654 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-26 02:04:27.330631818 +0000 2024-03-26T02:05:06.654 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T02:05:06.718 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-26T02:05:06.719 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-26T02:05:06.719 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.0481e-05 s, 5.7 MB/s 2024-03-26T02:05:06.720 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T02:05:06.775 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2024-03-26T02:05:06.830 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T02:05:06.830 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:06.830 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-26T02:05:06.830 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:06.830 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:06.830 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-26 02:05:01.056200415 +0000 2024-03-26T02:05:06.830 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-26 02:04:27.666627520 +0000 2024-03-26T02:05:06.830 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-26 02:04:27.666627520 +0000 2024-03-26T02:05:06.830 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-26 02:04:27.666627520 +0000 2024-03-26T02:05:06.831 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T02:05:06.895 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-26T02:05:06.895 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-26T02:05:06.895 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 8.8525e-05 s, 5.8 MB/s 2024-03-26T02:05:06.896 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T02:05:06.951 DEBUG:teuthology.orchestra.run.smithi052:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T02:05:07.091 INFO:teuthology.orchestra.run.smithi052.stdout:loop 2024-03-26T02:05:07.093 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_1... 2024-03-26T02:05:07.093 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T02:05:07.130 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-03-26T02:05:07.192 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_11 2024-03-26T02:05:07.234 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_2... 2024-03-26T02:05:07.234 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T02:05:07.314 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-03-26T02:05:07.353 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_21 2024-03-26T02:05:07.383 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_3... 2024-03-26T02:05:07.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T02:05:07.462 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-03-26T02:05:07.499 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_31 2024-03-26T02:05:07.540 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_4... 2024-03-26T02:05:07.541 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T02:05:07.618 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-03-26T02:05:07.656 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_41 2024-03-26T02:05:07.692 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:05:07.692 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:05:07.747 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme list 2024-03-26T02:05:07.812 INFO:teuthology.orchestra.run.smithi052.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T02:05:07.812 INFO:teuthology.orchestra.run.smithi052.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T02:05:07.812 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme0n1 PHMB7513010N480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-26T02:05:07.812 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme1n1 f7f1de61938e60e35fc2 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:07.812 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme2n1 e35b0736fab73bc61901 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:07.812 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme3n1 368e20502751d9c63f2c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:07.813 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme4n1 dd0e295881e725b924cd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:07.813 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T02:05:07.814 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:05:07.814 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/scratch_devs 2024-03-26T02:05:07.879 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-26T02:05:07.879 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:05:07.898 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T02:05:07.898 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-03-26T02:05:07.956 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T02:05:07.956 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:07.956 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-26T02:05:07.956 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:07.956 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:07.956 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-26 02:05:01.075806520 +0000 2024-03-26T02:05:07.956 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-26 02:04:24.328365736 +0000 2024-03-26T02:05:07.956 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-26 02:04:24.328365736 +0000 2024-03-26T02:05:07.957 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-26 02:04:24.328365736 +0000 2024-03-26T02:05:07.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T02:05:08.024 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-26T02:05:08.024 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-26T02:05:08.024 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000209815 s, 2.4 MB/s 2024-03-26T02:05:08.026 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T02:05:08.082 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-03-26T02:05:08.137 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T02:05:08.137 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:08.137 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T02:05:08.137 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:08.137 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:08.137 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-26 02:05:01.075806520 +0000 2024-03-26T02:05:08.137 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-26 02:04:24.675360456 +0000 2024-03-26T02:05:08.138 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-26 02:04:24.675360456 +0000 2024-03-26T02:05:08.138 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-26 02:04:24.675360456 +0000 2024-03-26T02:05:08.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T02:05:08.204 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-26T02:05:08.204 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-26T02:05:08.204 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000182078 s, 2.8 MB/s 2024-03-26T02:05:08.206 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T02:05:08.262 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-03-26T02:05:08.317 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T02:05:08.317 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:08.317 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-26T02:05:08.318 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:08.318 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:08.318 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-26 02:05:01.075806520 +0000 2024-03-26T02:05:08.318 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-26 02:04:25.053354704 +0000 2024-03-26T02:05:08.318 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-26 02:04:25.053354704 +0000 2024-03-26T02:05:08.318 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-26 02:04:25.053354704 +0000 2024-03-26T02:05:08.318 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T02:05:08.384 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-26T02:05:08.384 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-26T02:05:08.384 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000193276 s, 2.6 MB/s 2024-03-26T02:05:08.385 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T02:05:08.440 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-03-26T02:05:08.495 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T02:05:08.495 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:08.495 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T02:05:08.495 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:08.495 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:08.495 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-26 02:05:01.075806520 +0000 2024-03-26T02:05:08.495 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-26 02:04:25.398349453 +0000 2024-03-26T02:05:08.495 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-26 02:04:25.398349453 +0000 2024-03-26T02:05:08.495 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-26 02:04:25.398349453 +0000 2024-03-26T02:05:08.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T02:05:08.561 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-26T02:05:08.561 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-26T02:05:08.561 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000181673 s, 2.8 MB/s 2024-03-26T02:05:08.562 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T02:05:08.618 DEBUG:teuthology.orchestra.run.smithi165:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T02:05:08.760 INFO:teuthology.orchestra.run.smithi165.stdout:loop 2024-03-26T02:05:08.762 INFO:tasks.nvme_loop:Connecting nvme_loop smithi165:/dev/vg_nvme/lv_1... 2024-03-26T02:05:08.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T02:05:08.799 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-03-26T02:05:08.837 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/vg_nvme/lv_11 2024-03-26T02:05:08.867 INFO:tasks.nvme_loop:Connecting nvme_loop smithi165:/dev/vg_nvme/lv_2... 2024-03-26T02:05:08.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T02:05:08.946 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-03-26T02:05:08.984 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/vg_nvme/lv_21 2024-03-26T02:05:09.013 INFO:tasks.nvme_loop:Connecting nvme_loop smithi165:/dev/vg_nvme/lv_3... 2024-03-26T02:05:09.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T02:05:09.092 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-03-26T02:05:09.131 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/vg_nvme/lv_31 2024-03-26T02:05:09.168 INFO:tasks.nvme_loop:Connecting nvme_loop smithi165:/dev/vg_nvme/lv_4... 2024-03-26T02:05:09.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T02:05:09.246 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-03-26T02:05:09.284 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/vg_nvme/lv_41 2024-03-26T02:05:09.312 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-26T02:05:09.312 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:05:09.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo nvme list 2024-03-26T02:05:09.433 INFO:teuthology.orchestra.run.smithi165.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T02:05:09.433 INFO:teuthology.orchestra.run.smithi165.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T02:05:09.433 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/nvme0n1 CVFT623300CE400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-26T02:05:09.433 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/nvme1n1 f8f8be4974f01429b2fd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:09.434 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/nvme2n1 e3edd30f285bd9641707 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:09.434 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/nvme3n1 6959243787108a82da9e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:09.434 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/nvme4n1 7498ded24bdb4b1877e6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:09.435 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T02:05:09.435 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-26T02:05:09.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/scratch_devs 2024-03-26T02:05:09.502 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-26T02:05:09.502 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:05:09.521 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T02:05:09.521 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-03-26T02:05:09.579 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T02:05:09.579 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:09.579 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-26T02:05:09.579 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:09.579 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:09.579 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-26 02:05:01.146610201 +0000 2024-03-26T02:05:09.579 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-26 02:04:24.532255826 +0000 2024-03-26T02:05:09.580 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-26 02:04:24.532255826 +0000 2024-03-26T02:05:09.580 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-26 02:04:24.532255826 +0000 2024-03-26T02:05:09.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T02:05:09.648 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-26T02:05:09.648 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-26T02:05:09.648 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00024868 s, 2.1 MB/s 2024-03-26T02:05:09.649 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T02:05:09.707 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-03-26T02:05:09.763 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T02:05:09.764 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:09.764 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-26T02:05:09.764 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:09.764 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:09.764 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-26 02:05:01.146610201 +0000 2024-03-26T02:05:09.764 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-26 02:04:24.870249866 +0000 2024-03-26T02:05:09.764 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-26 02:04:24.870249866 +0000 2024-03-26T02:05:09.764 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-26 02:04:24.870249866 +0000 2024-03-26T02:05:09.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T02:05:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-26T02:05:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-26T02:05:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000213433 s, 2.4 MB/s 2024-03-26T02:05:09.832 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T02:05:09.889 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-03-26T02:05:09.945 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T02:05:09.945 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:09.945 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-26T02:05:09.945 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:09.945 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:09.945 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-26 02:05:01.146610201 +0000 2024-03-26T02:05:09.945 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-26 02:04:25.204243977 +0000 2024-03-26T02:05:09.945 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-26 02:04:25.204243977 +0000 2024-03-26T02:05:09.945 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-26 02:04:25.204243977 +0000 2024-03-26T02:05:09.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T02:05:10.011 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-26T02:05:10.011 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-26T02:05:10.011 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000247244 s, 2.1 MB/s 2024-03-26T02:05:10.013 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T02:05:10.069 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-03-26T02:05:10.125 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T02:05:10.125 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:05:10.125 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-26T02:05:10.125 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:05:10.125 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:05:10.125 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-26 02:05:01.146610201 +0000 2024-03-26T02:05:10.125 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-26 02:04:25.544237982 +0000 2024-03-26T02:05:10.125 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-26 02:04:25.544237982 +0000 2024-03-26T02:05:10.125 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-26 02:04:25.544237982 +0000 2024-03-26T02:05:10.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T02:05:10.191 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-26T02:05:10.191 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-26T02:05:10.191 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000162424 s, 3.2 MB/s 2024-03-26T02:05:10.192 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T02:05:10.248 DEBUG:teuthology.orchestra.run.smithi191:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-26T02:05:10.382 INFO:teuthology.orchestra.run.smithi191.stdout:loop 2024-03-26T02:05:10.383 INFO:tasks.nvme_loop:Connecting nvme_loop smithi191:/dev/vg_nvme/lv_1... 2024-03-26T02:05:10.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-26T02:05:10.435 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-03-26T02:05:10.473 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/vg_nvme/lv_11 2024-03-26T02:05:10.511 INFO:tasks.nvme_loop:Connecting nvme_loop smithi191:/dev/vg_nvme/lv_2... 2024-03-26T02:05:10.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-26T02:05:10.551 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-03-26T02:05:10.589 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/vg_nvme/lv_21 2024-03-26T02:05:10.619 INFO:tasks.nvme_loop:Connecting nvme_loop smithi191:/dev/vg_nvme/lv_3... 2024-03-26T02:05:10.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-26T02:05:10.698 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-03-26T02:05:10.736 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/vg_nvme/lv_31 2024-03-26T02:05:10.765 INFO:tasks.nvme_loop:Connecting nvme_loop smithi191:/dev/vg_nvme/lv_4... 2024-03-26T02:05:10.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-26T02:05:10.843 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-03-26T02:05:10.882 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/vg_nvme/lv_41 2024-03-26T02:05:10.917 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-26T02:05:10.917 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:05:10.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo nvme list 2024-03-26T02:05:11.037 INFO:teuthology.orchestra.run.smithi191.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-26T02:05:11.038 INFO:teuthology.orchestra.run.smithi191.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-26T02:05:11.038 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/nvme0n1 CVFT6303008M400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-26T02:05:11.038 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/nvme1n1 98a21a3322fec246b29c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:11.038 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/nvme2n1 6ab9f5db68774c1474be Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:11.038 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/nvme3n1 8d9fbb8547eb4597fdef Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:11.038 INFO:teuthology.orchestra.run.smithi191.stdout:/dev/nvme4n1 39375883a584f906e495 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-26T02:05:11.039 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T02:05:11.039 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-26T02:05:11.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/scratch_devs 2024-03-26T02:05:11.106 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T02:05:11.220 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'} 2024-03-26T02:05:11.221 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T02:05:11.221 INFO:tasks.cephadm:Cluster fsid is 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:05:11.221 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T02:05:11.221 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.52', 'mon.b': '172.21.15.165', 'mon.c': '172.21.15.191'} 2024-03-26T02:05:11.222 INFO:tasks.cephadm:First mon is mon.a on smithi052 2024-03-26T02:05:11.222 INFO:tasks.cephadm:First mgr is y 2024-03-26T02:05:11.222 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T02:05:11.222 DEBUG:teuthology.orchestra.run.smithi052:> sudo hostname $(hostname -s) 2024-03-26T02:05:11.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo hostname $(hostname -s) 2024-03-26T02:05:11.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname $(hostname -s) 2024-03-26T02:05:11.316 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-26T02:05:11.316 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T02:05:11.460 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54690/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1775-g5e70124b', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1775.g5e70124b'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-25 18:23:16.641218', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-26T02:05:11.710 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 5e70124b46c2a671efcaae4690cd9ea72670710a from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T02:05:11.712 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm 2024-03-26T02:05:11.712 DEBUG:teuthology.orchestra.run.smithi052:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T02:05:12.282 INFO:teuthology.orchestra.run.smithi052.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 02:05 /home/ubuntu/cephtest/cephadm 2024-03-26T02:05:12.282 DEBUG:teuthology.orchestra.run.smithi165:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T02:05:12.864 INFO:teuthology.orchestra.run.smithi165.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 02:05 /home/ubuntu/cephtest/cephadm 2024-03-26T02:05:12.864 DEBUG:teuthology.orchestra.run.smithi191:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T02:05:13.699 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 02:05 /home/ubuntu/cephtest/cephadm 2024-03-26T02:05:13.700 DEBUG:teuthology.orchestra.run.smithi052:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T02:05:13.718 DEBUG:teuthology.orchestra.run.smithi165:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T02:05:13.737 DEBUG:teuthology.orchestra.run.smithi191:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T02:05:13.770 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on all hosts... 2024-03-26T02:05:13.770 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T02:05:13.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T02:05:13.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T02:05:14.681 INFO:teuthology.orchestra.run.smithi165.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T02:05:14.728 INFO:teuthology.orchestra.run.smithi191.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T02:05:14.820 INFO:teuthology.orchestra.run.smithi052.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T02:05:46.597 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-03-26T02:05:46.598 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T02:05:46.598 INFO:teuthology.orchestra.run.smithi052.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T02:05:46.598 INFO:teuthology.orchestra.run.smithi052.stdout: "repo_digests": [ 2024-03-26T02:05:46.598 INFO:teuthology.orchestra.run.smithi052.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T02:05:46.598 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-03-26T02:05:46.598 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-03-26T02:06:00.200 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-03-26T02:06:00.200 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T02:06:00.200 INFO:teuthology.orchestra.run.smithi165.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T02:06:00.200 INFO:teuthology.orchestra.run.smithi165.stdout: "repo_digests": [ 2024-03-26T02:06:00.201 INFO:teuthology.orchestra.run.smithi165.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T02:06:00.201 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-03-26T02:06:00.201 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-03-26T02:06:02.998 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-26T02:06:02.998 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T02:06:02.999 INFO:teuthology.orchestra.run.smithi191.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T02:06:02.999 INFO:teuthology.orchestra.run.smithi191.stdout: "repo_digests": [ 2024-03-26T02:06:02.999 INFO:teuthology.orchestra.run.smithi191.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T02:06:02.999 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-26T02:06:02.999 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-26T02:06:03.022 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph 2024-03-26T02:06:03.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph 2024-03-26T02:06:03.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph 2024-03-26T02:06:03.122 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /etc/ceph 2024-03-26T02:06:03.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /etc/ceph 2024-03-26T02:06:03.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /etc/ceph 2024-03-26T02:06:03.210 INFO:tasks.cephadm:Writing seed config... 2024-03-26T02:06:03.212 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T02:06:03.212 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T02:06:03.212 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-26T02:06:03.212 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-26T02:06:03.212 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T02:06:03.212 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T02:06:03.212 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T02:06:03.212 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T02:06:03.212 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T02:06:03.212 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T02:06:03.213 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:06:03.213 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T02:06:03.229 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 46f2b2dc-eb15-11ee-95d0-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-26T02:06:03.230 DEBUG:teuthology.orchestra.run.smithi052:mon.a> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T02:06:03.272 DEBUG:teuthology.orchestra.run.smithi052:mgr.y> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T02:06:03.314 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T02:06:03.314 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a -v bootstrap --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.52 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:06:03.512 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:06:03.513 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a', '-v', 'bootstrap', '--fsid', '46f2b2dc-eb15-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.52', '--skip-admin-label'] 2024-03-26T02:06:03.538 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T02:06:03.538 INFO:teuthology.orchestra.run.smithi052.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T02:06:03.538 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying podman|docker is present... 2024-03-26T02:06:03.561 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T02:06:03.561 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying lvm2 is present... 2024-03-26T02:06:03.561 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying time synchronization is in place... 2024-03-26T02:06:03.568 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T02:06:03.568 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T02:06:03.574 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T02:06:03.574 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-26T02:06:03.580 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2024-03-26T02:06:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2024-03-26T02:06:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2024-03-26T02:06:03.586 INFO:teuthology.orchestra.run.smithi052.stdout:Repeating the final host check... 2024-03-26T02:06:03.609 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T02:06:03.609 INFO:teuthology.orchestra.run.smithi052.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T02:06:03.609 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl is present 2024-03-26T02:06:03.609 INFO:teuthology.orchestra.run.smithi052.stdout:lvcreate is present 2024-03-26T02:06:03.615 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T02:06:03.615 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T02:06:03.621 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T02:06:03.621 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-26T02:06:03.627 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2024-03-26T02:06:03.633 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2024-03-26T02:06:03.633 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2024-03-26T02:06:03.633 INFO:teuthology.orchestra.run.smithi052.stdout:Host looks OK 2024-03-26T02:06:03.633 INFO:teuthology.orchestra.run.smithi052.stdout:Cluster fsid: 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:06:03.633 INFO:teuthology.orchestra.run.smithi052.stdout:Acquiring lock 140410989095520 on /run/cephadm/46f2b2dc-eb15-11ee-95d0-87774f69a715.lock 2024-03-26T02:06:03.633 INFO:teuthology.orchestra.run.smithi052.stdout:Lock 140410989095520 acquired on /run/cephadm/46f2b2dc-eb15-11ee-95d0-87774f69a715.lock 2024-03-26T02:06:03.633 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying IP 172.21.15.52 port 3300 ... 2024-03-26T02:06:03.634 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying IP 172.21.15.52 port 6789 ... 2024-03-26T02:06:03.634 INFO:teuthology.orchestra.run.smithi052.stdout:Base mon IP(s) is [172.21.15.52:3300, 172.21.15.52:6789], mon addrv is [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2024-03-26T02:06:03.637 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.52 metric 100 2024-03-26T02:06:03.637 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.52 metric 100 2024-03-26T02:06:03.639 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T02:06:03.639 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-26T02:06:03.639 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-26T02:06:03.642 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T02:06:03.642 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T02:06:03.642 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T02:06:03.642 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-26T02:06:03.642 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15de/64 scope link noprefixroute 2024-03-26T02:06:03.642 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T02:06:03.643 INFO:teuthology.orchestra.run.smithi052.stdout:Mon IP `172.21.15.52` is in CIDR network `172.21.0.0/20` 2024-03-26T02:06:03.643 INFO:teuthology.orchestra.run.smithi052.stdout:Mon IP `172.21.15.52` is in CIDR network `172.21.0.0/20` 2024-03-26T02:06:03.643 INFO:teuthology.orchestra.run.smithi052.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T02:06:03.643 INFO:teuthology.orchestra.run.smithi052.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T02:06:03.643 INFO:teuthology.orchestra.run.smithi052.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T02:06:04.106 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T02:06:04.107 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T02:06:04.107 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T02:06:04.107 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:45ddbf13772d0d6535ecfdcaa58ecfc496e5849512dfbf602c5218332a216bcc 2024-03-26T02:06:04.107 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T02:06:04.107 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying config sha256:5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T02:06:04.107 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T02:06:05.052 INFO:teuthology.orchestra.run.smithi052.stdout:ceph: stdout ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T02:06:05.052 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph version: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T02:06:05.052 INFO:teuthology.orchestra.run.smithi052.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T02:06:06.091 INFO:teuthology.orchestra.run.smithi052.stdout:stat: stdout 167 167 2024-03-26T02:06:06.092 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial keys... 2024-03-26T02:06:07.155 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQCOLQJmxnD7GhAADApeIzumAA7sb6LZQ9eCgA== 2024-03-26T02:06:08.101 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQCPLQJmfaJdHhAAOskYj8e39jt4suY27C0bNQ== 2024-03-26T02:06:09.006 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQCQLQJmGNofHBAAo1RmtGOeDcGeHDugsnECOQ== 2024-03-26T02:06:09.007 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial monmap... 2024-03-26T02:06:09.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T02:06:09.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T02:06:09.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:06:09.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T02:06:09.971 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool for a [v2:172.21.15.52:3300,v1:172.21.15.52:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T02:06:09.971 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = quincy 2024-03-26T02:06:09.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: set fsid to 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:06:09.971 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T02:06:09.971 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:06:09.971 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mon... 2024-03-26T02:06:11.068 INFO:teuthology.orchestra.run.smithi052.stdout:create mon.a on 2024-03-26T02:06:11.523 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T02:06:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715.target. 2024-03-26T02:06:11.951 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715.target. 2024-03-26T02:06:12.894 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.a 2024-03-26T02:06:12.894 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T02:06:13.126 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715.target.wants/ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@.service. 2024-03-26T02:06:13.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:13 smithi052 podman[25365]: 2024-03-26 02:06:13.264048301 +0000 UTC m=+0.017498127 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T02:06:14.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:14 smithi052 podman[25365]: 2024-03-26 02:06:14.241984816 +0000 UTC m=+0.995434641 container create 5fc4751fcdf4233b3ab475ad4ae0f6e264e11c3c585e25cc321cd6e8c7335cf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T02:06:16.513 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:16 smithi052 podman[25365]: 2024-03-26 02:06:16.234650767 +0000 UTC m=+2.988100597 container init 5fc4751fcdf4233b3ab475ad4ae0f6e264e11c3c585e25cc321cd6e8c7335cf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T02:06:16.513 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:16 smithi052 podman[25365]: 2024-03-26 02:06:16.241234996 +0000 UTC m=+2.994684813 container start 5fc4751fcdf4233b3ab475ad4ae0f6e264e11c3c585e25cc321cd6e8c7335cf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T02:06:16.513 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:16 smithi052 ceph-mon[25402]: mkfs 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:06:16.513 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:16 smithi052 ceph-mon[25402]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T02:06:17.063 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T02:06:17.063 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-03-26T02:06:17.071 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T02:06:17.071 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-26T02:06:17.071 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-03-26T02:06:17.071 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T02:06:17.072 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon to start... 2024-03-26T02:06:17.072 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon... 2024-03-26T02:06:17.304 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:17 smithi052 bash[25365]: 5fc4751fcdf4233b3ab475ad4ae0f6e264e11c3c585e25cc321cd6e8c7335cf9 2024-03-26T02:06:17.304 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:17 smithi052 systemd[1]: Started Ceph mon.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:06:17.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:17 smithi052 ceph-mon[25402]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T02:06:17.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:17 smithi052 ceph-mon[25402]: monmap e1: 1 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T02:06:17.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:17 smithi052 ceph-mon[25402]: fsmap 2024-03-26T02:06:17.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:17 smithi052 ceph-mon[25402]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T02:06:17.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:17 smithi052 ceph-mon[25402]: mgrmap e1: no daemons active 2024-03-26T02:06:20.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:20 smithi052 ceph-mon[25402]: from='client.? 172.21.15.52:0/2539790971' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:06:21.158 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T02:06:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout id: 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:06:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T02:06:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout services: 2024-03-26T02:06:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 4s) 2024-03-26T02:06:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T02:06:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T02:06:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout data: 2024-03-26T02:06:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T02:06:21.159 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T02:06:21.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T02:06:21.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T02:06:21.160 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:21.160 INFO:teuthology.orchestra.run.smithi052.stdout:mon is available 2024-03-26T02:06:21.160 INFO:teuthology.orchestra.run.smithi052.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T02:06:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:21 smithi052 ceph-mon[25402]: from='client.? 172.21.15.52:0/1139531213' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T02:06:21.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:21 smithi052 ceph-mon[25402]: from='client.? 172.21.15.52:0/1139531213' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T02:06:22.288 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:22.289 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T02:06:22.290 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T02:06:22.290 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T02:06:22.290 INFO:teuthology.orchestra.run.smithi052.stdout:Generating new minimal ceph.conf... 2024-03-26T02:06:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:22 smithi052 ceph-mon[25402]: from='client.? 172.21.15.52:0/1122398859' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:23.526 INFO:teuthology.orchestra.run.smithi052.stdout:Restarting the monitor... 2024-03-26T02:06:23.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:23 smithi052 systemd[1]: Stopping Ceph mon.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:06:24.059 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:23 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a[25379]: 2024-03-26T02:06:23.803+0000 7fbe9e870700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:06:24.060 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:23 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a[25379]: 2024-03-26T02:06:23.803+0000 7fbe9e870700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T02:06:24.326 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:24 smithi052 podman[25735]: 2024-03-26 02:06:24.059522853 +0000 UTC m=+0.363342424 container died 5fc4751fcdf4233b3ab475ad4ae0f6e264e11c3c585e25cc321cd6e8c7335cf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T02:06:24.701 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:24 smithi052 podman[25735]: 2024-03-26 02:06:24.41716794 +0000 UTC m=+0.720987509 container cleanup 5fc4751fcdf4233b3ab475ad4ae0f6e264e11c3c585e25cc321cd6e8c7335cf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:06:24.701 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:24 smithi052 bash[25735]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a 2024-03-26T02:06:24.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:24 smithi052 podman[25752]: 2024-03-26 02:06:24.70134931 +0000 UTC m=+0.644723949 container remove 5fc4751fcdf4233b3ab475ad4ae0f6e264e11c3c585e25cc321cd6e8c7335cf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:06:24.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:24 smithi052 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T02:06:24.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:24 smithi052 systemd[1]: Stopped Ceph mon.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:06:24.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:24 smithi052 systemd[1]: Starting Ceph mon.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:06:25.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 podman[25850]: 2024-03-26 02:06:25.052188717 +0000 UTC m=+0.105665241 container create bd4ba28f38651b196a674b5ec4c2901c8459f91944ee784085c5ecd5d835221c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T02:06:25.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 podman[25850]: 2024-03-26 02:06:24.963892303 +0000 UTC m=+0.017368812 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T02:06:25.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 podman[25850]: 2024-03-26 02:06:25.270926735 +0000 UTC m=+0.324403219 container init bd4ba28f38651b196a674b5ec4c2901c8459f91944ee784085c5ecd5d835221c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, RELEASE=squid-5e70124, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T02:06:25.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 podman[25850]: 2024-03-26 02:06:25.275872021 +0000 UTC m=+0.329348503 container start bd4ba28f38651b196a674b5ec4c2901c8459f91944ee784085c5ecd5d835221c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-26T02:06:25.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T02:06:25.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T02:06:25.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: pidfile_write: ignore empty --pid-file 2024-03-26T02:06:25.303 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: load: jerasure load: lrc 2024-03-26T02:06:25.408 INFO:teuthology.orchestra.run.smithi052.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T02:06:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: RocksDB version: 7.9.2 2024-03-26T02:06:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Git sha 0 2024-03-26T02:06:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T02:06:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: DB SUMMARY 2024-03-26T02:06:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: DB Session ID: UAKJCO5J7N6JG8NHYKOE 2024-03-26T02:06:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: CURRENT file: CURRENT 2024-03-26T02:06:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84924 ; 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.error_if_exists: 0 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.create_if_missing: 0 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.paranoid_checks: 1 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.env: 0x55a6680c3ae0 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.info_log: 0x55a669411920 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.statistics: (nil) 2024-03-26T02:06:25.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.use_fsync: 0 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_log_file_size: 0 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.allow_fallocate: 1 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.use_direct_reads: 0 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.db_log_dir: 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.wal_dir: 2024-03-26T02:06:25.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T02:06:25.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T02:06:25.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T02:06:25.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T02:06:25.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T02:06:25.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T02:06:25.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T02:06:25.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T02:06:25.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.write_buffer_manager: 0x55a6693e41e0 2024-03-26T02:06:25.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.unordered_write: 0 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.row_cache: None 2024-03-26T02:06:25.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.wal_filter: None 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.two_write_queues: 0 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.wal_compression: 0 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.atomic_flush: 0 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.log_readahead_size: 0 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T02:06:25.717 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_background_jobs: 2 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_background_compactions: -1 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_subcompactions: 1 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T02:06:25.718 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_open_files: -1 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_background_flushes: -1 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Compression algorithms supported: 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: kZSTD supported: 0 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: kXpressCompression supported: 0 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: kZlibCompression supported: 1 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: kSnappyCompression supported: 1 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: kLZ4Compression supported: 1 2024-03-26T02:06:25.719 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: kBZip2Compression supported: 0 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.merge_operator: 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_filter: None 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a669411a20) 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout: cache_index_and_filter_blocks: 1 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T02:06:25.720 INFO:journalctl@ceph.mon.a.smithi052.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: pin_top_level_index_and_filter: 1 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_type: 0 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: data_block_index_type: 0 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_shortening: 1 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: checksum: 4 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: no_block_cache: 0 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache: 0x55a669415090 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_name: BinnedLRUCache 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_options: 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: capacity : 536870912 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: num_shard_bits : 4 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: strict_capacity_limit : 0 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: high_pri_pool_ratio: 0.000 2024-03-26T02:06:25.721 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_compressed: (nil) 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: persistent_cache: (nil) 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_size: 4096 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_size_deviation: 10 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_restart_interval: 16 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_block_restart_interval: 1 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: metadata_block_size: 4096 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: partition_filters: 0 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: use_delta_encoding: 1 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: filter_policy: bloomfilter 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: whole_key_filtering: 1 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: verify_compression: 0 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: read_amp_bytes_per_bit: 0 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: format_version: 5 2024-03-26T02:06:25.722 INFO:journalctl@ceph.mon.a.smithi052.stdout: enable_index_compression: 1 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_align: 0 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout: max_auto_readahead_size: 262144 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout: prepopulate_block_cache: 0 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout: initial_auto_readahead_size: 8192 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compression: NoCompression 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.num_levels: 7 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T02:06:25.723 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T02:06:25.724 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T02:06:25.725 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T02:06:25.726 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.table_properties_collectors: 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.inplace_update_support: 0 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.bloom_locality: 0 2024-03-26T02:06:25.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.max_successive_merges: 0 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.ttl: 2592000 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.enable_blob_files: false 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.min_blob_size: 0 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T02:06:25.728 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: eeb40724-297f-4019-b2f3-1574c6594fb4 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418785304893, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418785305728, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81438, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 258, "table_properties": {"data_size": 79607, "index_size": 241, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10057, "raw_average_key_size": 47, "raw_value_size": 73871, "raw_average_value_size": 346, "num_data_blocks": 11, "num_entries": 213, "num_filter_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711418785, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "eeb40724-297f-4019-b2f3-1574c6594fb4", "db_session_id": "UAKJCO5J7N6JG8NHYKOE", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418785305811, "job": 1, "event": "recovery_finished"} 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T02:06:25.729 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a669506000 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: DB pointer 0x55a6694f0000 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** DB Stats ** 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** Compaction Stats [default] ** 2024-03-26T02:06:25.730 INFO:journalctl@ceph.mon.a.smithi052.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: L0 2/0 81.36 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: Sum 2/0 81.36 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** Compaction Stats [default] ** 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T02:06:25.731 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative compaction: 0.00 GB write, 27.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval compaction: 0.00 GB write, 27.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout: Block cache BinnedLRUCache@0x55a669415090#7 capacity: 512.00 MB usage: 1.20 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: starting mon.a rank 0 at public addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] at bind addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: mon.a@-1(???) e1 preinit fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: mon.a@-1(???).mds e1 new map 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: mon.a@-1(???).mds e1 print_map 2024-03-26T02:06:25.732 INFO:journalctl@ceph.mon.a.smithi052.stdout: e1 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout: legacy client fscid: -1 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout: No filesystems configured 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: monmap e1: 1 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: fsmap 2024-03-26T02:06:25.733 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T02:06:25.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 ceph-mon[25889]: mgrmap e1: no daemons active 2024-03-26T02:06:25.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 bash[25850]: bd4ba28f38651b196a674b5ec4c2901c8459f91944ee784085c5ecd5d835221c 2024-03-26T02:06:25.734 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:25 smithi052 systemd[1]: Started Ceph mon.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:06:26.896 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T02:06:26.918 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:06:26.919 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mgr... 2024-03-26T02:06:26.919 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T02:06:26.919 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T02:06:27.166 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.y 2024-03-26T02:06:27.166 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T02:06:27.318 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715.target.wants/ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@.service. 2024-03-26T02:06:27.336 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:27 smithi052 systemd[1]: Starting Ceph mgr.y for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:06:27.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:27 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3208812690' entity='client.admin' 2024-03-26T02:06:27.648 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:27 smithi052 podman[26130]: 2024-03-26 02:06:27.545808204 +0000 UTC m=+0.110885808 container create 280496602fcc91698097fb9dbcbd52eb037881a123df67e9f5525ca7b16ed63e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-03-26T02:06:27.648 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:27 smithi052 podman[26130]: 2024-03-26 02:06:27.452743098 +0000 UTC m=+0.017820656 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T02:06:27.811 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T02:06:27.811 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-03-26T02:06:27.818 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T02:06:27.818 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-26T02:06:27.818 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-03-26T02:06:27.818 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T02:06:27.826 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T02:06:27.826 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-03-26T02:06:27.833 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T02:06:27.833 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-26T02:06:27.833 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-03-26T02:06:27.834 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T02:06:27.834 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr to start... 2024-03-26T02:06:27.834 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr... 2024-03-26T02:06:28.037 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:27 smithi052 podman[26130]: 2024-03-26 02:06:27.739516592 +0000 UTC m=+0.304594150 container init 280496602fcc91698097fb9dbcbd52eb037881a123df67e9f5525ca7b16ed63e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:06:28.037 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:27 smithi052 podman[26130]: 2024-03-26 02:06:27.745033714 +0000 UTC m=+0.310111271 container start 280496602fcc91698097fb9dbcbd52eb037881a123df67e9f5525ca7b16ed63e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-03-26T02:06:28.037 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:27 smithi052 bash[26130]: 280496602fcc91698097fb9dbcbd52eb037881a123df67e9f5525ca7b16ed63e 2024-03-26T02:06:28.037 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:27 smithi052 systemd[1]: Started Ceph mgr.y for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:06:28.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:28 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3043646863' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T02:06:28.645 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:28 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:28.315+0000 7f7d3873f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:06:28.645 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:28 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:28.415+0000 7f7d3873f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:06:28.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:28.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:06:28.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "46f2b2dc-eb15-11ee-95d0-87774f69a715", 2024-03-26T02:06:28.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T02:06:28.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T02:06:28.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T02:06:28.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T02:06:28.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:28.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T02:06:28.934 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T02:06:28.935 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T02:06:28.936 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T02:06:28.936 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T02:06:28.936 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T02:06:28.936 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T02:06:28.939 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:28.939 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T02:06:28.939 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T02:06:28.939 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T02:06:28.939 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T02:06:28.939 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T02:06:28.939 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T02:06:28.939 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T02:06:28.939 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T02:06:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:28.941 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:06:28.941 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:28.941 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T02:06:28.941 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:28.941 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T02:06:16.300027+0000", 2024-03-26T02:06:28.941 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:06:28.941 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:28.941 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T02:06:28.941 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:06:28.941 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (1/15)... 2024-03-26T02:06:29.210 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:29.081+0000 7f7d3873f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:06:29.210 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:29.175+0000 7f7d3873f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:06:29.710 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:29.318+0000 7f7d3873f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:06:29.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:29.416+0000 7f7d3873f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:06:29.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:29.501+0000 7f7d3873f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:06:30.067 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:29.769+0000 7f7d3873f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:06:30.067 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:29.881+0000 7f7d3873f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:06:30.067 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:29.969+0000 7f7d3873f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:06:30.067 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:30 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:30.066+0000 7f7d3873f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:06:30.460 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:30 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:30.244+0000 7f7d3873f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:06:31.375 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:31.256+0000 7f7d3873f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:06:31.683 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:31 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2019490688' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T02:06:31.683 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:31.394+0000 7f7d3873f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:06:31.950 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:31.743+0000 7f7d3873f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:06:31.951 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:31.834+0000 7f7d3873f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:06:31.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:31.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:06:31.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "46f2b2dc-eb15-11ee-95d0-87774f69a715", 2024-03-26T02:06:31.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T02:06:31.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T02:06:31.956 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T02:06:31.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T02:06:31.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:31.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T02:06:31.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T02:06:31.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-03-26T02:06:31.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:31.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T02:06:31.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T02:06:31.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:31.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-26T02:06:31.957 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T02:06:31.958 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T02:06:31.962 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:31.962 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T02:06:31.962 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T02:06:31.962 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T02:06:31.962 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T02:06:31.962 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T02:06:31.963 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T02:06:16.300027+0000", 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T02:06:31.965 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:06:31.965 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (2/15)... 2024-03-26T02:06:32.210 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:32 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:32.034+0000 7f7d3873f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:06:32.512 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:32 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:32.223+0000 7f7d3873f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:06:32.773 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:32 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:32.511+0000 7f7d3873f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:06:32.773 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:32 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:32.772+0000 7f7d3873f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:06:33.210 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:33 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:33.013+0000 7f7d3873f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:06:34.174 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:33 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:33.838+0000 7f7d3873f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:06:34.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:34 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/883351319' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T02:06:34.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:34 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:34.430+0000 7f7d3873f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:06:34.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:34 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:34.533+0000 7f7d3873f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:06:35.002 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "46f2b2dc-eb15-11ee-95d0-87774f69a715", 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:35.003 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T02:06:35.004 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T02:06:35.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T02:06:35.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T02:06:35.005 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T02:06:35.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:35.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T02:06:35.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T02:06:35.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T02:06:35.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T02:06:35.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T02:06:35.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T02:06:35.008 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T02:06:35.009 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T02:06:35.010 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:35.010 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:06:35.010 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:35.010 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T02:06:35.010 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:35.010 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T02:06:16.300027+0000", 2024-03-26T02:06:35.010 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:06:35.010 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:35.010 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T02:06:35.010 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:06:35.010 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (3/15)... 2024-03-26T02:06:35.210 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:34 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:34.755+0000 7f7d3873f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:06:35.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: Activating manager daemon y 2024-03-26T02:06:35.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: mgrmap e2: y(active, starting, since 0.00308016s) 2024-03-26T02:06:35.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: from='mgr.14100 172.21.15.52:0/704326767' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:06:35.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: from='mgr.14100 172.21.15.52:0/704326767' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:06:35.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: from='mgr.14100 172.21.15.52:0/704326767' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:06:35.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: from='mgr.14100 172.21.15.52:0/704326767' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:06:35.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: from='mgr.14100 172.21.15.52:0/704326767' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:06:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: Manager daemon y is now available 2024-03-26T02:06:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: from='mgr.14100 172.21.15.52:0/704326767' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:06:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: from='mgr.14100 172.21.15.52:0/704326767' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:06:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: from='mgr.14100 172.21.15.52:0/704326767' entity='mgr.y' 2024-03-26T02:06:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: from='mgr.14100 172.21.15.52:0/704326767' entity='mgr.y' 2024-03-26T02:06:35.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:35 smithi052 ceph-mon[25889]: from='mgr.14100 172.21.15.52:0/704326767' entity='mgr.y' 2024-03-26T02:06:37.130 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:36 smithi052 ceph-mon[25889]: mgrmap e3: y(active, since 1.00689s) 2024-03-26T02:06:38.062 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:37 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/569133618' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T02:06:38.331 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:38.331 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:06:38.331 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "46f2b2dc-eb15-11ee-95d0-87774f69a715", 2024-03-26T02:06:38.331 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T02:06:38.331 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T02:06:38.331 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T02:06:38.331 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T02:06:38.331 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T02:06:38.332 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T02:06:38.333 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T02:06:38.334 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:06:38.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:06:38.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:38.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T02:06:38.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:06:38.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T02:06:16.300027+0000", 2024-03-26T02:06:38.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:06:38.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:06:38.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T02:06:38.335 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:06:38.335 INFO:teuthology.orchestra.run.smithi052.stdout:mgr is available 2024-03-26T02:06:38.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:38 smithi052 ceph-mon[25889]: mgrmap e4: y(active, since 3s) 2024-03-26T02:06:39.472 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:39.472 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T02:06:39.472 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:06:39.472 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T02:06:39.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T02:06:39.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T02:06:39.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T02:06:39.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:39.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T02:06:39.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T02:06:39.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-26T02:06:39.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T02:06:39.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T02:06:39.473 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T02:06:39.473 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling cephadm module... 2024-03-26T02:06:39.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:39 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/104213555' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T02:06:39.896 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:39 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/104213555' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T02:06:41.115 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:40 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/1299206876' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T02:06:41.116 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:40 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ignoring --setuser ceph since I am not root 2024-03-26T02:06:41.116 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:40 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ignoring --setgroup ceph since I am not root 2024-03-26T02:06:41.366 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:41 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:41.350+0000 7f4d53536200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:06:41.626 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:41 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:41.454+0000 7f4d53536200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:06:41.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:41 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/1299206876' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T02:06:41.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:41 smithi052 ceph-mon[25889]: mgrmap e5: y(active, since 6s) 2024-03-26T02:06:42.362 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:42 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:42.122+0000 7f4d53536200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:06:42.362 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:42 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:42.218+0000 7f4d53536200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:06:42.362 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:42 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:42.361+0000 7f4d53536200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:06:42.710 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:42 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:42.461+0000 7f4d53536200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:06:42.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:42 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:42.547+0000 7f4d53536200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:06:42.733 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:06:42.733 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T02:06:42.733 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T02:06:42.733 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T02:06:42.733 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T02:06:42.733 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:06:42.734 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2024-03-26T02:06:42.734 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 5... 2024-03-26T02:06:43.027 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:42 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3188904717' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T02:06:43.027 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:42 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:42.821+0000 7f4d53536200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:06:43.027 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:42 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:42.934+0000 7f4d53536200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:06:43.027 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:43 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:43.026+0000 7f4d53536200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:06:43.305 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:43 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:43.125+0000 7f4d53536200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:06:43.305 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:43 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:43.304+0000 7f4d53536200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:06:44.710 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:44 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:44.284+0000 7f4d53536200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:06:44.710 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:44 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:44.417+0000 7f4d53536200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:06:45.050 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:44 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:44.758+0000 7f4d53536200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:06:45.050 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:44 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:44.849+0000 7f4d53536200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:06:45.050 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:45 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:45.049+0000 7f4d53536200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:06:45.460 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:45 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:45.239+0000 7f4d53536200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:06:45.797 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:45 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:45.531+0000 7f4d53536200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:06:45.797 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:45 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:45.796+0000 7f4d53536200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:06:46.210 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:46 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:46.039+0000 7f4d53536200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:06:47.210 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:46 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:46.861+0000 7f4d53536200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:06:47.549 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:47.444+0000 7f4d53536200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:06:47.841 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:47.548+0000 7f4d53536200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:06:47.841 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:06:47.775+0000 7f4d53536200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: Active manager daemon y restarted 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: Activating manager daemon y 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: mgrmap e6: y(active, starting, since 0.00411154s) 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: Manager daemon y is now available 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: Found migration_current of "None". Setting to last migration. 2024-03-26T02:06:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:06:48.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:06:48.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:47 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:06:49.102 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:48 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:49.102 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:48 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:49.102 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:48 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:49.102 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:48 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:06:49.103 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:48 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:06:49.103 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:48 smithi052 ceph-mon[25889]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T02:06:49.103 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:48 smithi052 ceph-mon[25889]: mgrmap e7: y(active, since 1.00519s) 2024-03-26T02:06:49.103 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:48 smithi052 ceph-mon[25889]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T02:06:49.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:06:49.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T02:06:49.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T02:06:49.372 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:06:49.372 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 5 is available 2024-03-26T02:06:49.372 INFO:teuthology.orchestra.run.smithi052.stdout:Setting orchestrator backend to cephadm... 2024-03-26T02:06:50.341 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:50 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:06:50.341 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:50 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:06:50.341 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:50 smithi052 ceph-mon[25889]: [26/Mar/2024:02:06:49] ENGINE Bus STARTING 2024-03-26T02:06:50.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:50 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:50.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:50 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:06:50.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:50 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:51.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:51 smithi052 ceph-mon[25889]: [26/Mar/2024:02:06:49] ENGINE Serving on https://172.21.15.52:7150 2024-03-26T02:06:51.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:51 smithi052 ceph-mon[25889]: [26/Mar/2024:02:06:49] ENGINE Serving on http://172.21.15.52:8765 2024-03-26T02:06:51.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:51 smithi052 ceph-mon[25889]: [26/Mar/2024:02:06:49] ENGINE Bus STARTED 2024-03-26T02:06:51.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:51 smithi052 ceph-mon[25889]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:51.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:51 smithi052 ceph-mon[25889]: mgrmap e8: y(active, since 2s) 2024-03-26T02:06:51.809 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T02:06:51.809 INFO:teuthology.orchestra.run.smithi052.stdout:Generating ssh key... 2024-03-26T02:06:52.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-mon[25889]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:53.155 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: Generating public/private rsa key pair. 2024-03-26T02:06:53.155 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: Your identification has been saved in /tmp/tmpvyrpwhmh/key. 2024-03-26T02:06:53.155 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: Your public key has been saved in /tmp/tmpvyrpwhmh/key.pub. 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: The key fingerprint is: 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: SHA256:1+MizI3ZsjgNqzbFsV/gK/IwWg0+OupvxtlYIJh258Q ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: The key's randomart image is: 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: +---[RSA 3072]----+ 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: | | 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: | | 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: |.. . | 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: |oo o E. . . | 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: |. o *. +S.. o | 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: | . ==o.*.. . | 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: | . @..=Bo+ . | 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: | .X+=+.++ . | 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: |oo*+.=oo. | 2024-03-26T02:06:53.156 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:06:52 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: +----[SHA256]-----+ 2024-03-26T02:06:53.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:53 smithi052 ceph-mon[25889]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:53.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:53 smithi052 ceph-mon[25889]: Generating ssh key... 2024-03-26T02:06:53.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:53 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:06:53.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:53 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:06:54.579 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpjNLSe1tkUO0X7QpnW0mqAL6cE9YKtbEpOCZ7xN0oflG7PEQ2Fp8wtRg9MMmWzhZNcPzKYHfbGvCgNvvf/R5dVXfBkBhDTimqAQJ902jzyykMZ87H7RvZAcmzwJw9bEhOu8E9Ovuspt4Rr+Ew2i1EhP9/ZylMgEGxzv3la9/6a6e7jhbhitOEvpLRrQh5SW/gnzDfJfoOYA67YFXckoOGnfWe11r+JtCzzmG4mzzsWNLny1tYQroYCwE4gHEPc/cZzNZXcBCbSeWtHLFFpcui1qq00QmHse+1B4GUf3gYpUhWylcCO/ICU9O2xMlPDfjncHKUadjeb8Nj+mk88ImYWOd9FNbduf16kuwZ8vCGofi1FIoZt/X8GiOA9xQbaVIq6J7Jelw42VGedhwS2HGHGJQo/46k4nPgmQmDM1Bc9tDvCfF+JS5ic7mJJMc5zIPIpNOueaX/a3SeSh0itAKnpPEYGstLd0VdD5YSX6juQqBe36IZlFHwYwGfPAIo6Y8= ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:06:54.579 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:06:54.579 INFO:teuthology.orchestra.run.smithi052.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T02:06:54.579 INFO:teuthology.orchestra.run.smithi052.stdout:Adding host smithi052... 2024-03-26T02:06:55.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:54 smithi052 ceph-mon[25889]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:56.203 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:55 smithi052 ceph-mon[25889]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi052", "addr": "172.21.15.52", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:57.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:06:56 smithi052 ceph-mon[25889]: Deploying cephadm binary to smithi052 2024-03-26T02:07:00.093 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Added host 'smithi052' with addr '172.21.15.52' 2024-03-26T02:07:00.093 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying unmanaged mon service... 2024-03-26T02:07:00.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:00 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:07:00.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:00 smithi052 ceph-mon[25889]: Added host smithi052 2024-03-26T02:07:00.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:00 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:01.326 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T02:07:01.326 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying unmanaged mgr service... 2024-03-26T02:07:02.036 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:01 smithi052 ceph-mon[25889]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:02.037 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:01 smithi052 ceph-mon[25889]: Saving service mon spec with placement count:5 2024-03-26T02:07:02.037 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:01 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:07:02.818 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T02:07:03.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:03 smithi052 ceph-mon[25889]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:03.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:03 smithi052 ceph-mon[25889]: Saving service mgr spec with placement count:2 2024-03-26T02:07:03.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:03 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:07:04.651 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:04 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/837524371' entity='client.admin' 2024-03-26T02:07:04.651 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:04 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:07:04.651 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:04 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:07:05.791 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling the dashboard module... 2024-03-26T02:07:06.057 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:05 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2725533594' entity='client.admin' 2024-03-26T02:07:07.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:07 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:07:07.929 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:07 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:07:07.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:07 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:07:07.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:07 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' 2024-03-26T02:07:07.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:07 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/64542349' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T02:07:07.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:07 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]: dispatch 2024-03-26T02:07:07.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:07 smithi052 ceph-mon[25889]: from='mgr.14120 172.21.15.52:0/2467637333' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]': finished 2024-03-26T02:07:07.930 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:07 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ignoring --setuser ceph since I am not root 2024-03-26T02:07:07.930 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:07 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ignoring --setgroup ceph since I am not root 2024-03-26T02:07:08.462 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:08 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:08.208+0000 7f8f69611200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:07:08.462 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:08 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:08.312+0000 7f8f69611200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:07:08.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:08 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/64542349' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T02:07:08.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:08 smithi052 ceph-mon[25889]: mgrmap e9: y(active, since 19s) 2024-03-26T02:07:09.232 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:08 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:08.990+0000 7f8f69611200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:07:09.232 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:09 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:09.087+0000 7f8f69611200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:07:09.232 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:09 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:09.231+0000 7f8f69611200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:07:09.594 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:09 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:09.330+0000 7f8f69611200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:07:09.594 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:09 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:09.416+0000 7f8f69611200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:07:09.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:07:09.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T02:07:09.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T02:07:09.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T02:07:09.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T02:07:09.597 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:07:09.597 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2024-03-26T02:07:09.597 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 9... 2024-03-26T02:07:09.895 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:09 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2020036722' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T02:07:09.896 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:09 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:09.688+0000 7f8f69611200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:07:09.896 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:09 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:09.802+0000 7f8f69611200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:07:09.896 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:09 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:09.894+0000 7f8f69611200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:07:10.174 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:09 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:09.994+0000 7f8f69611200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:07:10.174 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:10 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:10.173+0000 7f8f69611200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:07:11.460 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:11 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:11.157+0000 7f8f69611200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:07:11.460 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:11 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:11.291+0000 7f8f69611200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:07:11.920 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:11 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:11.627+0000 7f8f69611200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:07:11.960 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:11 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:11.719+0000 7f8f69611200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:07:11.960 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:11 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:11.920+0000 7f8f69611200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:07:12.210 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:12 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:12.109+0000 7f8f69611200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:07:12.662 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:12 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:12.401+0000 7f8f69611200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:07:12.662 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:12 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:12.661+0000 7f8f69611200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:07:12.962 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:12 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:12.903+0000 7f8f69611200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:07:14.210 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:13 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:13.736+0000 7f8f69611200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:07:14.663 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:14.331+0000 7f8f69611200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:07:14.663 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:14.435+0000 7f8f69611200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:07:14.663 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:07:14.662+0000 7f8f69611200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:07:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: Active manager daemon y restarted 2024-03-26T02:07:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: Activating manager daemon y 2024-03-26T02:07:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T02:07:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: mgrmap e10: y(active, starting, since 0.00333106s) 2024-03-26T02:07:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:07:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:07:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:07:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:07:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:07:14.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: Manager daemon y is now available 2024-03-26T02:07:14.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:14.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:14.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:15.959 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:15 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:07:15.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:15 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:07:15.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:15 smithi052 ceph-mon[25889]: [26/Mar/2024:02:07:15] ENGINE Bus STARTING 2024-03-26T02:07:15.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:15 smithi052 ceph-mon[25889]: [26/Mar/2024:02:07:15] ENGINE Serving on https://172.21.15.52:7150 2024-03-26T02:07:15.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:15 smithi052 ceph-mon[25889]: [26/Mar/2024:02:07:15] ENGINE Serving on http://172.21.15.52:8765 2024-03-26T02:07:15.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:15 smithi052 ceph-mon[25889]: [26/Mar/2024:02:07:15] ENGINE Bus STARTED 2024-03-26T02:07:15.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:15 smithi052 ceph-mon[25889]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T02:07:15.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:15 smithi052 ceph-mon[25889]: mgrmap e11: y(active, since 1.00597s) 2024-03-26T02:07:16.138 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:07:16.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T02:07:16.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T02:07:16.139 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:07:16.139 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 9 is available 2024-03-26T02:07:16.139 INFO:teuthology.orchestra.run.smithi052.stdout:Generating a dashboard self-signed certificate... 2024-03-26T02:07:16.797 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:16 smithi052 ceph-mon[25889]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T02:07:16.797 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:16 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:16.797 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:16 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:07:16.797 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:16 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:16.798 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:16 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]: dispatch 2024-03-26T02:07:17.702 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T02:07:17.703 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial admin user... 2024-03-26T02:07:17.809 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:17 smithi052 ceph-mon[25889]: mgrmap e12: y(active, since 2s) 2024-03-26T02:07:17.809 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:17 smithi052 ceph-mon[25889]: Deploying daemon agent.smithi052 on smithi052 2024-03-26T02:07:17.809 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:17 smithi052 ceph-mon[25889]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:17.809 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:17.809 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:19.300 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$FRF.biE5Nv9sbOl6fLFIF.XklmCo0dur.OBPZnfl7fPQaMso6K27W", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711418838, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T02:07:19.300 INFO:teuthology.orchestra.run.smithi052.stdout:Fetching dashboard port number... 2024-03-26T02:07:19.691 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:19 smithi052 ceph-mon[25889]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:19.692 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:19 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:20.506 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T02:07:20.517 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T02:07:20.517 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-03-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-03-26T02:07:20.524 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T02:07:20.526 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph Dashboard is now available at: 2024-03-26T02:07:20.526 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: URL: https://smithi052.front.sepia.ceph.com:8443/ 2024-03-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: User: admin 2024-03-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: Password: 9hsnjmeic3 2024-03-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout:Saving cluster configuration to /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config directory 2024-03-26T02:07:20.527 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling autotune for osd_memory_target 2024-03-26T02:07:20.761 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:20 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3005389586' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T02:07:22.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:22 smithi052 ceph-mon[25889]: mgrmap e13: y(active, since 6s) 2024-03-26T02:07:22.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:22 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2423669637' entity='client.admin' 2024-03-26T02:07:23.067 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T02:07:23.067 INFO:teuthology.orchestra.run.smithi052.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T02:07:23.067 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:23.067 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:07:23.067 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:23.067 INFO:teuthology.orchestra.run.smithi052.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T02:07:23.067 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:23.067 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T02:07:23.067 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:23.067 INFO:teuthology.orchestra.run.smithi052.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T02:07:23.067 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:23.068 INFO:teuthology.orchestra.run.smithi052.stdout: ceph telemetry on 2024-03-26T02:07:23.068 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:23.068 INFO:teuthology.orchestra.run.smithi052.stdout:For more information see: 2024-03-26T02:07:23.068 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:23.068 INFO:teuthology.orchestra.run.smithi052.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T02:07:23.068 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:23.068 INFO:teuthology.orchestra.run.smithi052.stdout:Bootstrap complete. 2024-03-26T02:07:23.109 INFO:tasks.cephadm:Fetching config... 2024-03-26T02:07:23.110 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:07:23.110 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T02:07:23.125 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T02:07:23.125 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:07:23.125 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T02:07:23.179 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T02:07:23.179 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:07:23.179 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T02:07:23.330 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T02:07:23.330 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:07:23.330 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T02:07:23.394 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T02:07:23.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpjNLSe1tkUO0X7QpnW0mqAL6cE9YKtbEpOCZ7xN0oflG7PEQ2Fp8wtRg9MMmWzhZNcPzKYHfbGvCgNvvf/R5dVXfBkBhDTimqAQJ902jzyykMZ87H7RvZAcmzwJw9bEhOu8E9Ovuspt4Rr+Ew2i1EhP9/ZylMgEGxzv3la9/6a6e7jhbhitOEvpLRrQh5SW/gnzDfJfoOYA67YFXckoOGnfWe11r+JtCzzmG4mzzsWNLny1tYQroYCwE4gHEPc/cZzNZXcBCbSeWtHLFFpcui1qq00QmHse+1B4GUf3gYpUhWylcCO/ICU9O2xMlPDfjncHKUadjeb8Nj+mk88ImYWOd9FNbduf16kuwZ8vCGofi1FIoZt/X8GiOA9xQbaVIq6J7Jelw42VGedhwS2HGHGJQo/46k4nPgmQmDM1Bc9tDvCfF+JS5ic7mJJMc5zIPIpNOueaX/a3SeSh0itAKnpPEYGstLd0VdD5YSX6juQqBe36IZlFHwYwGfPAIo6Y8= ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T02:07:23.532 INFO:teuthology.orchestra.run.smithi052.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpjNLSe1tkUO0X7QpnW0mqAL6cE9YKtbEpOCZ7xN0oflG7PEQ2Fp8wtRg9MMmWzhZNcPzKYHfbGvCgNvvf/R5dVXfBkBhDTimqAQJ902jzyykMZ87H7RvZAcmzwJw9bEhOu8E9Ovuspt4Rr+Ew2i1EhP9/ZylMgEGxzv3la9/6a6e7jhbhitOEvpLRrQh5SW/gnzDfJfoOYA67YFXckoOGnfWe11r+JtCzzmG4mzzsWNLny1tYQroYCwE4gHEPc/cZzNZXcBCbSeWtHLFFpcui1qq00QmHse+1B4GUf3gYpUhWylcCO/ICU9O2xMlPDfjncHKUadjeb8Nj+mk88ImYWOd9FNbduf16kuwZ8vCGofi1FIoZt/X8GiOA9xQbaVIq6J7Jelw42VGedhwS2HGHGJQo/46k4nPgmQmDM1Bc9tDvCfF+JS5ic7mJJMc5zIPIpNOueaX/a3SeSh0itAKnpPEYGstLd0VdD5YSX6juQqBe36IZlFHwYwGfPAIo6Y8= ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:07:23.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpjNLSe1tkUO0X7QpnW0mqAL6cE9YKtbEpOCZ7xN0oflG7PEQ2Fp8wtRg9MMmWzhZNcPzKYHfbGvCgNvvf/R5dVXfBkBhDTimqAQJ902jzyykMZ87H7RvZAcmzwJw9bEhOu8E9Ovuspt4Rr+Ew2i1EhP9/ZylMgEGxzv3la9/6a6e7jhbhitOEvpLRrQh5SW/gnzDfJfoOYA67YFXckoOGnfWe11r+JtCzzmG4mzzsWNLny1tYQroYCwE4gHEPc/cZzNZXcBCbSeWtHLFFpcui1qq00QmHse+1B4GUf3gYpUhWylcCO/ICU9O2xMlPDfjncHKUadjeb8Nj+mk88ImYWOd9FNbduf16kuwZ8vCGofi1FIoZt/X8GiOA9xQbaVIq6J7Jelw42VGedhwS2HGHGJQo/46k4nPgmQmDM1Bc9tDvCfF+JS5ic7mJJMc5zIPIpNOueaX/a3SeSh0itAKnpPEYGstLd0VdD5YSX6juQqBe36IZlFHwYwGfPAIo6Y8= ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T02:07:23.592 INFO:teuthology.orchestra.run.smithi165.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpjNLSe1tkUO0X7QpnW0mqAL6cE9YKtbEpOCZ7xN0oflG7PEQ2Fp8wtRg9MMmWzhZNcPzKYHfbGvCgNvvf/R5dVXfBkBhDTimqAQJ902jzyykMZ87H7RvZAcmzwJw9bEhOu8E9Ovuspt4Rr+Ew2i1EhP9/ZylMgEGxzv3la9/6a6e7jhbhitOEvpLRrQh5SW/gnzDfJfoOYA67YFXckoOGnfWe11r+JtCzzmG4mzzsWNLny1tYQroYCwE4gHEPc/cZzNZXcBCbSeWtHLFFpcui1qq00QmHse+1B4GUf3gYpUhWylcCO/ICU9O2xMlPDfjncHKUadjeb8Nj+mk88ImYWOd9FNbduf16kuwZ8vCGofi1FIoZt/X8GiOA9xQbaVIq6J7Jelw42VGedhwS2HGHGJQo/46k4nPgmQmDM1Bc9tDvCfF+JS5ic7mJJMc5zIPIpNOueaX/a3SeSh0itAKnpPEYGstLd0VdD5YSX6juQqBe36IZlFHwYwGfPAIo6Y8= ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:07:23.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpjNLSe1tkUO0X7QpnW0mqAL6cE9YKtbEpOCZ7xN0oflG7PEQ2Fp8wtRg9MMmWzhZNcPzKYHfbGvCgNvvf/R5dVXfBkBhDTimqAQJ902jzyykMZ87H7RvZAcmzwJw9bEhOu8E9Ovuspt4Rr+Ew2i1EhP9/ZylMgEGxzv3la9/6a6e7jhbhitOEvpLRrQh5SW/gnzDfJfoOYA67YFXckoOGnfWe11r+JtCzzmG4mzzsWNLny1tYQroYCwE4gHEPc/cZzNZXcBCbSeWtHLFFpcui1qq00QmHse+1B4GUf3gYpUhWylcCO/ICU9O2xMlPDfjncHKUadjeb8Nj+mk88ImYWOd9FNbduf16kuwZ8vCGofi1FIoZt/X8GiOA9xQbaVIq6J7Jelw42VGedhwS2HGHGJQo/46k4nPgmQmDM1Bc9tDvCfF+JS5ic7mJJMc5zIPIpNOueaX/a3SeSh0itAKnpPEYGstLd0VdD5YSX6juQqBe36IZlFHwYwGfPAIo6Y8= ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T02:07:23.655 INFO:teuthology.orchestra.run.smithi191.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpjNLSe1tkUO0X7QpnW0mqAL6cE9YKtbEpOCZ7xN0oflG7PEQ2Fp8wtRg9MMmWzhZNcPzKYHfbGvCgNvvf/R5dVXfBkBhDTimqAQJ902jzyykMZ87H7RvZAcmzwJw9bEhOu8E9Ovuspt4Rr+Ew2i1EhP9/ZylMgEGxzv3la9/6a6e7jhbhitOEvpLRrQh5SW/gnzDfJfoOYA67YFXckoOGnfWe11r+JtCzzmG4mzzsWNLny1tYQroYCwE4gHEPc/cZzNZXcBCbSeWtHLFFpcui1qq00QmHse+1B4GUf3gYpUhWylcCO/ICU9O2xMlPDfjncHKUadjeb8Nj+mk88ImYWOd9FNbduf16kuwZ8vCGofi1FIoZt/X8GiOA9xQbaVIq6J7Jelw42VGedhwS2HGHGJQo/46k4nPgmQmDM1Bc9tDvCfF+JS5ic7mJJMc5zIPIpNOueaX/a3SeSh0itAKnpPEYGstLd0VdD5YSX6juQqBe36IZlFHwYwGfPAIo6Y8= ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:07:23.670 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T02:07:24.087 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:07:24.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:24 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:24.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:24 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:24.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:24 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:24.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:24 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:24.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:24 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:24.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:24 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:24.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:24 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:24.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:24 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:24.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:24 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:24.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:24 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:24.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:24 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:25.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:27.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:26 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3223695063' entity='client.admin' 2024-03-26T02:07:27.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:26 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:27.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:26 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:27.897 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T02:07:27.897 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T02:07:28.322 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:07:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:29.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:31.413 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:31.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:31.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:31.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:31.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:31.414 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:31.442 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi165 2024-03-26T02:07:31.442 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-26T02:07:31.442 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.conf 2024-03-26T02:07:31.460 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-26T02:07:31.460 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:07:31.518 INFO:tasks.cephadm:Adding host smithi165 to orchestrator... 2024-03-26T02:07:31.518 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch host add smithi165 2024-03-26T02:07:31.772 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:07:32.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:32 smithi052 ceph-mon[25889]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:32.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:32 smithi052 ceph-mon[25889]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-26T02:07:32.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:32 smithi052 ceph-mon[25889]: Updating smithi052:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:07:32.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:32 smithi052 ceph-mon[25889]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:07:32.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:32 smithi052 ceph-mon[25889]: Updating smithi052:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:07:32.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:32 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:32.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:32 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:32.836 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:32 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:34.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:33 smithi052 ceph-mon[25889]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:35.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:35 smithi052 ceph-mon[25889]: Deploying cephadm binary to smithi165 2024-03-26T02:07:35.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:35 smithi052 ceph-mon[25889]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:37.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:37 smithi052 ceph-mon[25889]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:38.248 INFO:teuthology.orchestra.run.smithi052.stdout:Added host 'smithi165' with addr '172.21.15.165' 2024-03-26T02:07:38.998 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T02:07:39.238 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:07:39.524 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:39 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:39.524 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:39 smithi052 ceph-mon[25889]: Added host smithi165 2024-03-26T02:07:39.524 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:39 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:39.524 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:39 smithi052 ceph-mon[25889]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:39.524 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:39 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:40.541 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:40 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:40.541 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:40 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:40.541 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:40 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:07:40.541 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:40 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:40.541 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:40 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:40.541 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:40 smithi052 ceph-mon[25889]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-26T02:07:40.541 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:40 smithi052 ceph-mon[25889]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:07:40.541 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:40 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:40.541 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:40 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:40.541 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:40 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-26T02:07:40.542 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:40 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]': finished 2024-03-26T02:07:41.108 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:41.108 INFO:teuthology.orchestra.run.smithi052.stdout:[{"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}, {"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}] 2024-03-26T02:07:41.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:41 smithi052 ceph-mon[25889]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:07:41.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:41 smithi052 ceph-mon[25889]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:07:41.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:41 smithi052 ceph-mon[25889]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:41.695 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi191 2024-03-26T02:07:41.696 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-26T02:07:41.696 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.conf 2024-03-26T02:07:41.713 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-26T02:07:41.713 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:07:41.770 INFO:tasks.cephadm:Adding host smithi191 to orchestrator... 2024-03-26T02:07:41.771 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch host add smithi191 2024-03-26T02:07:42.003 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:07:42.432 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:42 smithi052 ceph-mon[25889]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:07:43.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: Deploying daemon agent.smithi165 on smithi165 2024-03-26T02:07:43.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:43.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:43.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:43.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:43.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:43.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:43.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:43.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:43.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:43.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:45.137 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:44 smithi052 ceph-mon[25889]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:45.138 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:44 smithi052 ceph-mon[25889]: Deploying cephadm binary to smithi191 2024-03-26T02:07:45.138 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:44 smithi052 ceph-mon[25889]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:46.068 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:45 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:47 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:47 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:47 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:47 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:47 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:47.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:47 smithi052 ceph-mon[25889]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:48.922 INFO:teuthology.orchestra.run.smithi052.stdout:Added host 'smithi191' with addr '172.21.15.191' 2024-03-26T02:07:49.155 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:48 smithi052 ceph-mon[25889]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:49.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T02:07:49.858 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:07:50.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:50.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:49 smithi052 ceph-mon[25889]: Added host smithi191 2024-03-26T02:07:50.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:50.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:50.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:50.984 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:07:50.985 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]': finished 2024-03-26T02:07:51.670 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:07:51.670 INFO:teuthology.orchestra.run.smithi052.stdout:[{"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}, {"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}, {"addr": "172.21.15.191", "hostname": "smithi191", "labels": [], "status": ""}] 2024-03-26T02:07:52.190 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T02:07:52.190 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T02:07:52.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:51 smithi052 ceph-mon[25889]: Deploying daemon agent.smithi191 on smithi191 2024-03-26T02:07:52.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:51 smithi052 ceph-mon[25889]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:07:52.425 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:07:53.646 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:53.647 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:53.648 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:54.573 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2024-03-26T02:07:54.886 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:54 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2279696488' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T02:07:55.179 INFO:tasks.cephadm:Adding mon.a on smithi052 2024-03-26T02:07:55.179 INFO:tasks.cephadm:Adding mon.b on smithi165 2024-03-26T02:07:55.180 INFO:tasks.cephadm:Adding mon.c on smithi191 2024-03-26T02:07:55.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi052:172.21.15.52=a;smithi165:172.21.15.165=b;smithi191:172.21.15.191=c' 2024-03-26T02:07:55.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:55 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2279696488' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T02:07:55.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:55 smithi052 ceph-mon[25889]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:07:55.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:55 smithi052 ceph-mon[25889]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:55.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:55.962 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:07:57.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:56 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:57.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:56 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:57.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:56 smithi052 ceph-mon[25889]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:57.755 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mon update... 2024-03-26T02:07:58.304 DEBUG:teuthology.orchestra.run.smithi165:mon.b> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T02:07:58.307 DEBUG:teuthology.orchestra.run.smithi191:mon.c> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T02:07:58.309 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T02:07:58.310 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T02:07:58.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:58 smithi052 ceph-mon[25889]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi165:172.21.15.165=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:58.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:58 smithi052 ceph-mon[25889]: Saving service mon spec with placement smithi052:172.21.15.52=a;smithi165:172.21.15.165=b;smithi191:172.21.15.191=c;count:3 2024-03-26T02:07:58.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:58 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:58.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:58 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:58.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:58 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:58.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:58 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:58.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:58 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:07:58.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:58 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:07:58.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:58 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:58.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:58 smithi052 ceph-mon[25889]: Deploying daemon mon.c on smithi191 2024-03-26T02:07:58.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:07:58 smithi052 ceph-mon[25889]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:07:59.167 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:08:01.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:00 smithi052 ceph-mon[25889]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:02.105 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T02:08:02.105 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":1,"fsid":"46f2b2dc-eb15-11ee-95d0-87774f69a715","modified":"2024-03-26T02:06:09.366978Z","created":"2024-03-26T02:06:09.366978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T02:08:02.107 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 1 2024-03-26T02:08:02.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:02 smithi052 ceph-mon[25889]: from='client.? 172.21.15.191:0/27425522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:02 smithi191 podman[27663]: 2024-03-26 02:08:02.851299446 +0000 UTC m=+0.018387669 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:08:03.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:03 smithi052 ceph-mon[25889]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 podman[27663]: 2024-03-26 02:08:03.120412817 +0000 UTC m=+0.287501038 container init 3c438a495cec85c07e9f66b706c7374222c3e288580a630c9fdc94e3dc9d45c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T02:08:03.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 podman[27663]: 2024-03-26 02:08:03.125644322 +0000 UTC m=+0.292732543 container start 3c438a495cec85c07e9f66b706c7374222c3e288580a630c9fdc94e3dc9d45c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124) 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pidfile_write: ignore empty --pid-file 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: load: jerasure load: lrc 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: RocksDB version: 7.9.2 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Git sha 0 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: DB SUMMARY 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: DB Session ID: 3ZH932UOLXQ15H5MV2X6 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: CURRENT file: CURRENT 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T02:08:03.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.error_if_exists: 0 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.create_if_missing: 0 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.paranoid_checks: 1 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.env: 0x559c75abeae0 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.info_log: 0x559c77553920 2024-03-26T02:08:03.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.statistics: (nil) 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.use_fsync: 0 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_log_file_size: 0 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.allow_fallocate: 1 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.use_direct_reads: 0 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T02:08:03.540 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.db_log_dir: 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.wal_dir: 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.write_buffer_manager: 0x559c775261e0 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T02:08:03.541 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.unordered_write: 0 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.row_cache: None 2024-03-26T02:08:03.542 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.wal_filter: None 2024-03-26T02:08:03.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T02:08:03.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T02:08:03.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.two_write_queues: 0 2024-03-26T02:08:03.544 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.wal_compression: 0 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.atomic_flush: 0 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.log_readahead_size: 0 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T02:08:03.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_background_jobs: 2 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_background_compactions: -1 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_subcompactions: 1 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_open_files: -1 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T02:08:03.546 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_background_flushes: -1 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Compression algorithms supported: 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: kZSTD supported: 0 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: kXpressCompression supported: 0 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: kZlibCompression supported: 1 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: kSnappyCompression supported: 1 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: kLZ4Compression supported: 1 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: kBZip2Compression supported: 0 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-26T02:08:03.547 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.merge_operator: 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_filter: None 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559c77553a20) 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: cache_index_and_filter_blocks: 1 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: pin_top_level_index_and_filter: 1 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_type: 0 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: data_block_index_type: 0 2024-03-26T02:08:03.548 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_shortening: 1 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: checksum: 4 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: no_block_cache: 0 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache: 0x559c77557090 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_name: BinnedLRUCache 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_options: 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: capacity : 536870912 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: num_shard_bits : 4 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: strict_capacity_limit : 0 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: high_pri_pool_ratio: 0.000 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_compressed: (nil) 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: persistent_cache: (nil) 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_size: 4096 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_size_deviation: 10 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_restart_interval: 16 2024-03-26T02:08:03.549 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_block_restart_interval: 1 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: metadata_block_size: 4096 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: partition_filters: 0 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: use_delta_encoding: 1 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: filter_policy: bloomfilter 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: whole_key_filtering: 1 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: verify_compression: 0 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: read_amp_bytes_per_bit: 0 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: format_version: 5 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: enable_index_compression: 1 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_align: 0 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: max_auto_readahead_size: 262144 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: prepopulate_block_cache: 0 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: initial_auto_readahead_size: 8192 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T02:08:03.550 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compression: NoCompression 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.num_levels: 7 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:08:03.551 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T02:08:03.552 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T02:08:03.553 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.table_properties_collectors: 2024-03-26T02:08:03.554 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.inplace_update_support: 0 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.bloom_locality: 0 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.max_successive_merges: 0 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.ttl: 2592000 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T02:08:03.555 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.enable_blob_files: false 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.min_blob_size: 0 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d8e32c34-7924-425d-bfa5-414f38ab8abd 2024-03-26T02:08:03.556 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418883155599, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418883156080, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711418883, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d8e32c34-7924-425d-bfa5-414f38ab8abd", "db_session_id": "3ZH932UOLXQ15H5MV2X6", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418883156160, "job": 1, "event": "recovery_finished"} 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559c77648000 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: DB pointer 0x559c77632000 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** DB Stats ** 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T02:08:03.557 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** Compaction Stats [default] ** 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-03-26T02:08:03.558 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** Compaction Stats [default] ** 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:08:03.559 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: Block cache BinnedLRUCache@0x559c77557090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: using public_addr v2:172.21.15.191:0/0 -> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: starting mon.c rank -1 at public addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] at bind addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(???) e0 preinit fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).mds e1 new map 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: e1 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T02:08:03.560 INFO:journalctl@ceph.mon.c.smithi191.stdout: legacy client fscid: -1 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout: No filesystems configured 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.561 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Added host smithi165 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]': finished 2024-03-26T02:08:03.562 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Deploying daemon agent.smithi165 on smithi165 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Deploying cephadm binary to smithi191 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.564 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Added host smithi191 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:08:03.565 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]': finished 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Deploying daemon agent.smithi191 on smithi191 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:03.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.567 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:03.567 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:03.567 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:03.567 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.567 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.569 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.569 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.569 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2279696488' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T02:08:03.569 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2279696488' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi165:172.21.15.165=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Saving service mon spec with placement smithi052:172.21.15.52=a;smithi165:172.21.15.165=b;smithi191:172.21.15.191=c;count:3 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:03.570 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:03.571 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:03.571 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: Deploying daemon mon.c on smithi191 2024-03-26T02:08:03.571 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.571 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.571 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: from='client.? 172.21.15.191:0/27425522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:03.571 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:03.571 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 ceph-mon[27701]: mon.c@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-26T02:08:03.571 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 bash[27663]: 3c438a495cec85c07e9f66b706c7374222c3e288580a630c9fdc94e3dc9d45c7 2024-03-26T02:08:03.571 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:03 smithi191 systemd[1]: Started Ceph mon.c for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:08:03.787 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T02:08:03.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T02:08:04.020 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:08:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:08:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: mon.a calling monitor election 2024-03-26T02:08:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: mon.c calling monitor election 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: monmap e2: 2 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: fsmap 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:08:08.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: mgrmap e13: y(active, since 53s) 2024-03-26T02:08:08.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: overall HEALTH_OK 2024-03-26T02:08:08.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.693 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T02:08:08.693 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":2,"fsid":"46f2b2dc-eb15-11ee-95d0-87774f69a715","modified":"2024-03-26T02:08:03.178987Z","created":"2024-03-26T02:06:09.366978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T02:08:08.695 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 2 2024-03-26T02:08:08.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:08:08.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: mon.a calling monitor election 2024-03-26T02:08:08.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:08.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: mon.c calling monitor election 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: monmap e2: 2 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: fsmap 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: mgrmap e13: y(active, since 53s) 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: overall HEALTH_OK 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:08.713 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:09 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:09 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:09 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:09 smithi191 ceph-mon[27701]: Deploying daemon mon.b on smithi165 2024-03-26T02:08:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:09 smithi191 ceph-mon[27701]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:09 smithi191 ceph-mon[27701]: from='client.? 172.21.15.191:0/2959709478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:09.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:09 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:09.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:09 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:09.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:09 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:09.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:09 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:09.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:09 smithi052 ceph-mon[25889]: Deploying daemon mon.b on smithi165 2024-03-26T02:08:09.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:09 smithi052 ceph-mon[25889]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:09.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:09 smithi052 ceph-mon[25889]: from='client.? 172.21.15.191:0/2959709478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:09.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:09 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:10.266 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T02:08:10.266 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T02:08:10.512 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:08:10.967 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:10 smithi191 ceph-mon[27701]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:11.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:10 smithi052 ceph-mon[25889]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 podman[27590]: 2024-03-26 02:08:12.155672991 +0000 UTC m=+0.119147331 container create 9eb5c805c486e165023bae6308f2167fa59a0d5a566d82c0f6819ac90537d13e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T02:08:12.354 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T02:08:12.354 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":2,"fsid":"46f2b2dc-eb15-11ee-95d0-87774f69a715","modified":"2024-03-26T02:08:03.178987Z","created":"2024-03-26T02:06:09.366978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T02:08:12.358 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 2 2024-03-26T02:08:12.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 podman[27590]: 2024-03-26 02:08:12.357922333 +0000 UTC m=+0.321396673 container init 9eb5c805c486e165023bae6308f2167fa59a0d5a566d82c0f6819ac90537d13e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-b, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:08:12.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 podman[27590]: 2024-03-26 02:08:12.363781709 +0000 UTC m=+0.327256047 container start 9eb5c805c486e165023bae6308f2167fa59a0d5a566d82c0f6819ac90537d13e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pidfile_write: ignore empty --pid-file 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: load: jerasure load: lrc 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: RocksDB version: 7.9.2 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Git sha 0 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: DB SUMMARY 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: DB Session ID: 4GF1YDJ0IOG6WTSSIP7W 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: CURRENT file: CURRENT 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-26T02:08:12.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-26T02:08:12.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.error_if_exists: 0 2024-03-26T02:08:12.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.create_if_missing: 0 2024-03-26T02:08:12.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.paranoid_checks: 1 2024-03-26T02:08:12.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T02:08:12.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T02:08:12.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T02:08:12.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.env: 0x55cf80a37ae0 2024-03-26T02:08:12.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T02:08:12.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.info_log: 0x55cf81d41920 2024-03-26T02:08:12.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T02:08:12.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.statistics: (nil) 2024-03-26T02:08:12.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.use_fsync: 0 2024-03-26T02:08:12.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_log_file_size: 0 2024-03-26T02:08:12.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.allow_fallocate: 1 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.use_direct_reads: 0 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.db_log_dir: 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.wal_dir: 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T02:08:12.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.write_buffer_manager: 0x55cf81d141e0 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T02:08:12.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.unordered_write: 0 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.row_cache: None 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.wal_filter: None 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.two_write_queues: 0 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.wal_compression: 0 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.atomic_flush: 0 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T02:08:12.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.log_readahead_size: 0 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_background_jobs: 2 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_background_compactions: -1 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_subcompactions: 1 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T02:08:12.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_open_files: -1 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_background_flushes: -1 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Compression algorithms supported: 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: kZSTD supported: 0 2024-03-26T02:08:12.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: kXpressCompression supported: 0 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: kZlibCompression supported: 1 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: kSnappyCompression supported: 1 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: kLZ4Compression supported: 1 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: kBZip2Compression supported: 0 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.merge_operator: 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_filter: None 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T02:08:12.712 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cf81d41a20) 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: cache_index_and_filter_blocks: 1 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: pin_top_level_index_and_filter: 1 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: index_type: 0 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: data_block_index_type: 0 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: index_shortening: 1 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: checksum: 4 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: no_block_cache: 0 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache: 0x55cf81d45090 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache_name: BinnedLRUCache 2024-03-26T02:08:12.713 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache_options: 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: capacity : 536870912 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: num_shard_bits : 4 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: strict_capacity_limit : 0 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: high_pri_pool_ratio: 0.000 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache_compressed: (nil) 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: persistent_cache: (nil) 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_size: 4096 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_size_deviation: 10 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_restart_interval: 16 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: index_block_restart_interval: 1 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: metadata_block_size: 4096 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: partition_filters: 0 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: use_delta_encoding: 1 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: filter_policy: bloomfilter 2024-03-26T02:08:12.714 INFO:journalctl@ceph.mon.b.smithi165.stdout: whole_key_filtering: 1 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout: verify_compression: 0 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout: read_amp_bytes_per_bit: 0 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout: format_version: 5 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout: enable_index_compression: 1 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_align: 0 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout: max_auto_readahead_size: 262144 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout: prepopulate_block_cache: 0 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout: initial_auto_readahead_size: 8192 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compression: NoCompression 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T02:08:12.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.num_levels: 7 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T02:08:12.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T02:08:12.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T02:08:12.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.table_properties_collectors: 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.inplace_update_support: 0 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T02:08:12.719 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.bloom_locality: 0 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.max_successive_merges: 0 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.ttl: 2592000 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.enable_blob_files: false 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.min_blob_size: 0 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T02:08:12.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8f8a299f-b9e4-4c17-b5c2-34b98c656d8e 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418892404669, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418892405085, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711418892, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8f8a299f-b9e4-4c17-b5c2-34b98c656d8e", "db_session_id": "4GF1YDJ0IOG6WTSSIP7W", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418892405165, "job": 1, "event": "recovery_finished"} 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T02:08:12.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cf81e36000 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: DB pointer 0x55cf81e20000 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** DB Stats ** 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** Compaction Stats [default] ** 2024-03-26T02:08:12.722 INFO:journalctl@ceph.mon.b.smithi165.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** Compaction Stats [default] ** 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T02:08:12.723 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout: Block cache BinnedLRUCache@0x55cf81d45090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: using public_addr v2:172.21.15.165:0/0 -> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: starting mon.b rank -1 at public addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] at bind addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:08:12.724 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(???) e0 preinit fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).mds e1 new map 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout: e1 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout: legacy client fscid: -1 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout: No filesystems configured 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T02:08:12.725 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Added host smithi165 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:12.726 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]': finished 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Deploying daemon agent.smithi165 on smithi165 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Deploying cephadm binary to smithi191 2024-03-26T02:08:12.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Added host smithi191 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]': finished 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Deploying daemon agent.smithi191 on smithi191 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2279696488' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2279696488' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi165:172.21.15.165=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Saving service mon spec with placement smithi052:172.21.15.52=a;smithi165:172.21.15.165=b;smithi191:172.21.15.191=c;count:3 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Deploying daemon mon.c on smithi191 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='client.? 172.21.15.191:0/27425522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.a calling monitor election 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.c calling monitor election 2024-03-26T02:08:12.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: monmap e2: 2 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: fsmap 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mgrmap e13: y(active, since 53s) 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: overall HEALTH_OK 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.734 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: Deploying daemon mon.b on smithi165 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='client.? 172.21.15.191:0/2959709478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: from='client.? 172.21.15.191:0/2217934477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 ceph-mon[27627]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 bash[27590]: 9eb5c805c486e165023bae6308f2167fa59a0d5a566d82c0f6819ac90537d13e 2024-03-26T02:08:12.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:12 smithi165 systemd[1]: Started Ceph mon.b for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:08:12.736 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:12 smithi052 ceph-mon[25889]: from='client.? 172.21.15.191:0/2217934477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:12.770 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:12 smithi191 ceph-mon[27701]: from='client.? 172.21.15.191:0/2217934477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:13.793 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T02:08:13.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T02:08:15.581 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:08:17.674 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: mon.c calling monitor election 2024-03-26T02:08:17.674 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:08:17.674 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: mon.a calling monitor election 2024-03-26T02:08:17.674 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.674 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:17.674 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:17.674 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: fsmap 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: mgrmap e13: y(active, since 62s) 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T02:08:17.675 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T02:08:17.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T02:08:17.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: mon.b (rank 2) addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] is down (out of quorum) 2024-03-26T02:08:17.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.676 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: mon.c calling monitor election 2024-03-26T02:08:17.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: mon.a calling monitor election 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T02:08:17.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: fsmap 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: mgrmap e13: y(active, since 62s) 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: mon.b (rank 2) addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] is down (out of quorum) 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:17.991 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-26T02:08:17.991 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":3,"fsid":"46f2b2dc-eb15-11ee-95d0-87774f69a715","modified":"2024-03-26T02:08:12.429300Z","created":"2024-03-26T02:06:09.366978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T02:08:17.993 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 3 2024-03-26T02:08:18.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:18.714 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='client.? 172.21.15.191:0/1418952858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:18.715 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:08:18.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:18.716 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:18 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:18.731 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:18.731 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:18.731 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:18.731 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='client.? 172.21.15.191:0/1418952858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:18.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:18 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:19.003 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T02:08:19.003 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T02:08:19.448 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: mon.c calling monitor election 2024-03-26T02:08:19.448 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:08:19.448 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: mon.a calling monitor election 2024-03-26T02:08:19.448 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:19.448 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:08:19.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:19.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:19.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:19.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:19.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:19.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:19.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:19.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:19.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T02:08:19.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T02:08:19.450 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: fsmap 2024-03-26T02:08:19.450 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:08:19.450 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: mgrmap e13: y(active, since 62s) 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: mon.b (rank 2) addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] is down (out of quorum) 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='client.? 172.21.15.191:0/1418952858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:19.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:19.890 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:08:20.011 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: mon.b calling monitor election 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: mon.b calling monitor election 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: mon.a calling monitor election 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: mon.c calling monitor election 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: fsmap 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: mgrmap e13: y(active, since 64s) 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: Cluster is now healthy 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: overall HEALTH_OK 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:20.012 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:19 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: mon.b calling monitor election 2024-03-26T02:08:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: mon.b calling monitor election 2024-03-26T02:08:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: mon.a calling monitor election 2024-03-26T02:08:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: mon.c calling monitor election 2024-03-26T02:08:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:08:20.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T02:08:20.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: fsmap 2024-03-26T02:08:20.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:08:20.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: mgrmap e13: y(active, since 64s) 2024-03-26T02:08:20.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T02:08:20.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: Cluster is now healthy 2024-03-26T02:08:20.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: overall HEALTH_OK 2024-03-26T02:08:20.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:20.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:19 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:20.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: mon.b calling monitor election 2024-03-26T02:08:20.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: mon.b calling monitor election 2024-03-26T02:08:20.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: mon.a calling monitor election 2024-03-26T02:08:20.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: mon.c calling monitor election 2024-03-26T02:08:20.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:08:20.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T02:08:20.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: fsmap 2024-03-26T02:08:20.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:08:20.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: mgrmap e13: y(active, since 64s) 2024-03-26T02:08:20.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T02:08:20.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: Cluster is now healthy 2024-03-26T02:08:20.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: overall HEALTH_OK 2024-03-26T02:08:20.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:20.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:19 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:20 smithi191 ceph-mon[27701]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:08:21.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:21.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:08:21.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:21.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:20 smithi191 ceph-mon[27701]: Reconfiguring daemon mon.b on smithi165 2024-03-26T02:08:21.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:21.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.032 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.033 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:20 smithi191 ceph-mon[27701]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:21.152 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.152 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:20 smithi052 ceph-mon[25889]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:08:21.152 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:21.152 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:08:21.152 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:21.152 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:20 smithi052 ceph-mon[25889]: Reconfiguring daemon mon.b on smithi165 2024-03-26T02:08:21.152 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:21.152 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.152 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.152 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.153 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:20 smithi052 ceph-mon[25889]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:21.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:20 smithi165 ceph-mon[27627]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:08:21.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:21.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:08:21.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:21.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:20 smithi165 ceph-mon[27627]: Reconfiguring daemon mon.b on smithi165 2024-03-26T02:08:21.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:08:21.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.160 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:21.160 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:20 smithi165 ceph-mon[27627]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:21.603 INFO:teuthology.orchestra.run.smithi052.stdout:# minimal ceph.conf for 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:08:21.604 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2024-03-26T02:08:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:08:21.604 INFO:teuthology.orchestra.run.smithi052.stdout: mon_host = [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-26T02:08:22.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:22 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:22 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:22 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:22 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:22 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:22 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:22 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/813549574' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:22.145 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T02:08:22.145 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:08:22.145 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T02:08:22.175 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:08:22.176 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:08:22.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:22 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:22 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:22 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:22 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:22 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:22 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:22 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/813549574' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:22.243 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-26T02:08:22.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T02:08:22.273 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-26T02:08:22.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:08:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:22 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:22 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:22 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:22 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:22 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:22 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:22.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:22 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/813549574' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:22.340 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-26T02:08:22.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T02:08:22.371 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-26T02:08:22.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:08:22.439 INFO:tasks.cephadm:Adding mgr.y on smithi052 2024-03-26T02:08:22.439 INFO:tasks.cephadm:Adding mgr.x on smithi165 2024-03-26T02:08:22.439 INFO:tasks.cephadm:Adding mgr.z on smithi191 2024-03-26T02:08:22.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi052=y;smithi165=x;smithi191=z' 2024-03-26T02:08:22.716 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:08:23.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:23 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:23.403 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:23 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:23.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:23 smithi191 ceph-mon[27701]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:08:23.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:23 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:23.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:23 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:08:23.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:23 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:23.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:23 smithi191 ceph-mon[27701]: Reconfiguring daemon mon.c on smithi191 2024-03-26T02:08:23.404 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:23 smithi191 ceph-mon[27701]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:23.562 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:23 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:23.562 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:23 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:23.562 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:23 smithi052 ceph-mon[25889]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:08:23.562 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:23 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:23.562 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:23 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:08:23.562 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:23 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:23.563 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:23 smithi052 ceph-mon[25889]: Reconfiguring daemon mon.c on smithi191 2024-03-26T02:08:23.563 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:23 smithi052 ceph-mon[25889]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:23.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:23 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:23.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:23 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:23.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:23 smithi165 ceph-mon[27627]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:08:23.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:23 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:08:23.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:23 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:08:23.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:23 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:23.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:23 smithi165 ceph-mon[27627]: Reconfiguring daemon mon.c on smithi191 2024-03-26T02:08:23.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:23 smithi165 ceph-mon[27627]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:25.405 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mgr update... 2024-03-26T02:08:25.948 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.948 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.948 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:25.948 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:25.948 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:25.948 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:25.948 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.948 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.948 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:25.949 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:25.949 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:25.949 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.949 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:08:25.949 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:08:25.949 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:08:25.949 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:25.950 DEBUG:teuthology.orchestra.run.smithi165:mgr.x> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T02:08:25.953 DEBUG:teuthology.orchestra.run.smithi191:mgr.z> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.z.service 2024-03-26T02:08:25.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:08:25.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:25.957 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T02:08:25.957 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:08:25.957 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:08:25.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:08:25.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:25.973 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T02:08:25.974 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/nvme1n1 2024-03-26T02:08:26.029 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/nvme1n1 2024-03-26T02:08:26.029 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:26.029 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-26T02:08:26.029 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:26.029 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:26.029 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-26 02:08:25.249825029 +0000 2024-03-26T02:08:26.030 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-26 02:05:07.239316409 +0000 2024-03-26T02:08:26.030 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-26 02:05:07.239316409 +0000 2024-03-26T02:08:26.030 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-26 02:05:07.232316499 +0000 2024-03-26T02:08:26.030 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T02:08:26.098 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-26T02:08:26.098 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-26T02:08:26.099 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000128006 s, 4.0 MB/s 2024-03-26T02:08:26.100 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T02:08:26.156 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/nvme2n1 2024-03-26T02:08:26.211 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/nvme2n1 2024-03-26T02:08:26.211 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:26.211 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-26T02:08:26.211 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:26.211 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:26.211 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-26 02:08:25.257824927 +0000 2024-03-26T02:08:26.211 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-26 02:05:07.391314460 +0000 2024-03-26T02:08:26.212 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-26 02:05:07.391314460 +0000 2024-03-26T02:08:26.212 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-26 02:05:07.381314589 +0000 2024-03-26T02:08:26.212 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T02:08:26.276 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-26T02:08:26.276 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-26T02:08:26.276 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.00011069 s, 4.6 MB/s 2024-03-26T02:08:26.277 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T02:08:26.333 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/nvme3n1 2024-03-26T02:08:26.387 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/nvme3n1 2024-03-26T02:08:26.388 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:26.388 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-26T02:08:26.388 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:26.388 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:26.388 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-26 02:08:25.266824813 +0000 2024-03-26T02:08:26.388 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-26 02:05:07.543316806 +0000 2024-03-26T02:08:26.388 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-26 02:05:07.543316806 +0000 2024-03-26T02:08:26.388 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-26 02:05:07.538316626 +0000 2024-03-26T02:08:26.388 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T02:08:26.452 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-26T02:08:26.452 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-26T02:08:26.453 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000106958 s, 4.8 MB/s 2024-03-26T02:08:26.454 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T02:08:26.509 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/nvme4n1 2024-03-26T02:08:26.564 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/nvme4n1 2024-03-26T02:08:26.564 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:26.564 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-26T02:08:26.564 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:26.564 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:26.564 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-26 02:08:25.274824711 +0000 2024-03-26T02:08:26.564 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-26 02:05:07.696322306 +0000 2024-03-26T02:08:26.564 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-26 02:05:07.696322306 +0000 2024-03-26T02:08:26.564 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-26 02:05:07.690322091 +0000 2024-03-26T02:08:26.565 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T02:08:26.629 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-26T02:08:26.630 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-26T02:08:26.630 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000108736 s, 4.7 MB/s 2024-03-26T02:08:26.631 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T02:08:26.686 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-26T02:08:26.686 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:08:26.704 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T02:08:26.705 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/nvme1n1 2024-03-26T02:08:26.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:26 smithi052 ceph-mon[25889]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=y;smithi165=x;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:26.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:26 smithi052 ceph-mon[25889]: Saving service mgr spec with placement smithi052=y;smithi165=x;smithi191=z;count:3 2024-03-26T02:08:26.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:26 smithi052 ceph-mon[25889]: Deploying daemon mgr.z on smithi191 2024-03-26T02:08:26.761 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/nvme1n1 2024-03-26T02:08:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:26.762 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-26T02:08:26.762 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:26.762 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:26.762 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-26 02:08:20.119664868 +0000 2024-03-26T02:08:26.762 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-26 02:05:08.878562637 +0000 2024-03-26T02:08:26.762 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-26 02:05:08.878562637 +0000 2024-03-26T02:08:26.762 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-26 02:05:08.866562818 +0000 2024-03-26T02:08:26.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T02:08:26.832 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-26T02:08:26.832 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-26T02:08:26.832 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000252903 s, 2.0 MB/s 2024-03-26T02:08:26.833 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T02:08:26.891 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/nvme2n1 2024-03-26T02:08:26.948 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/nvme2n1 2024-03-26T02:08:26.948 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:26.948 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-26T02:08:26.948 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:26.948 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:26.948 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-26 02:08:20.130664702 +0000 2024-03-26T02:08:26.948 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-26 02:05:09.022560459 +0000 2024-03-26T02:08:26.948 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-26 02:05:09.022560459 +0000 2024-03-26T02:08:26.948 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-26 02:05:09.011560626 +0000 2024-03-26T02:08:26.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T02:08:26.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:26 smithi165 ceph-mon[27627]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=y;smithi165=x;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:26.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:26 smithi165 ceph-mon[27627]: Saving service mgr spec with placement smithi052=y;smithi165=x;smithi191=z;count:3 2024-03-26T02:08:26.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:26 smithi165 ceph-mon[27627]: Deploying daemon mgr.z on smithi191 2024-03-26T02:08:26.979 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-26T02:08:26.979 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-26T02:08:26.979 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000243378 s, 2.1 MB/s 2024-03-26T02:08:26.980 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T02:08:26.988 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:26 smithi191 ceph-mon[27701]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=y;smithi165=x;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:26.988 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:26 smithi191 ceph-mon[27701]: Saving service mgr spec with placement smithi052=y;smithi165=x;smithi191=z;count:3 2024-03-26T02:08:26.988 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:26 smithi191 ceph-mon[27701]: Deploying daemon mgr.z on smithi191 2024-03-26T02:08:27.037 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/nvme3n1 2024-03-26T02:08:27.094 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/nvme3n1 2024-03-26T02:08:27.094 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:27.094 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-26T02:08:27.094 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:27.094 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:27.094 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-26 02:08:20.142664520 +0000 2024-03-26T02:08:27.094 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-26 02:05:09.175558145 +0000 2024-03-26T02:08:27.094 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-26 02:05:09.175558145 +0000 2024-03-26T02:08:27.094 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-26 02:05:09.166558281 +0000 2024-03-26T02:08:27.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T02:08:27.161 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-26T02:08:27.162 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-26T02:08:27.162 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000244645 s, 2.1 MB/s 2024-03-26T02:08:27.163 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T02:08:27.220 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/nvme4n1 2024-03-26T02:08:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/nvme4n1 2024-03-26T02:08:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:27.276 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-26T02:08:27.276 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:27.276 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:27.276 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-26 02:08:20.154664338 +0000 2024-03-26T02:08:27.276 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-26 02:05:09.321555935 +0000 2024-03-26T02:08:27.276 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-26 02:05:09.321555935 +0000 2024-03-26T02:08:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-26 02:05:09.310556102 +0000 2024-03-26T02:08:27.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T02:08:27.342 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-26T02:08:27.342 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-26T02:08:27.343 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000191137 s, 2.7 MB/s 2024-03-26T02:08:27.344 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T02:08:27.400 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-26T02:08:27.400 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:08:27.416 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-26T02:08:27.417 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/nvme1n1 2024-03-26T02:08:27.472 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/nvme1n1 2024-03-26T02:08:27.472 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:27.472 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 763 Links: 1 Device type: 103,2 2024-03-26T02:08:27.472 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:27.472 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:27.472 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-26 02:08:20.255820203 +0000 2024-03-26T02:08:27.472 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-26 02:05:10.519145922 +0000 2024-03-26T02:08:27.472 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-26 02:05:10.519145922 +0000 2024-03-26T02:08:27.472 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-26 02:05:10.510146081 +0000 2024-03-26T02:08:27.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-26T02:08:27.537 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:27 smithi191 systemd[1]: Starting Ceph mgr.z for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:08:27.542 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-26T02:08:27.542 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-26T02:08:27.542 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000286971 s, 1.8 MB/s 2024-03-26T02:08:27.543 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-26T02:08:27.599 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/nvme2n1 2024-03-26T02:08:27.654 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/nvme2n1 2024-03-26T02:08:27.655 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:27.655 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-26T02:08:27.655 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:27.655 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:27.655 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-26 02:08:20.266820011 +0000 2024-03-26T02:08:27.655 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-26 02:05:10.628144007 +0000 2024-03-26T02:08:27.655 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-26 02:05:10.628144007 +0000 2024-03-26T02:08:27.655 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-26 02:05:10.617144201 +0000 2024-03-26T02:08:27.655 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-26T02:08:27.720 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-26T02:08:27.721 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-26T02:08:27.721 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000230427 s, 2.2 MB/s 2024-03-26T02:08:27.722 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-26T02:08:27.779 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/nvme3n1 2024-03-26T02:08:27.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:27 smithi191 ceph-mon[27701]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:27.790 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:27 smithi191 podman[30264]: 2024-03-26 02:08:27.564840541 +0000 UTC m=+0.018111559 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:08:27.790 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:27 smithi191 podman[30264]: 2024-03-26 02:08:27.664817263 +0000 UTC m=+0.118088282 container create 1267993bed80a6a8833a2fd7ff26adb694c5ec954286fe7ee064074a6859446a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-03-26T02:08:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/nvme3n1 2024-03-26T02:08:27.805 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:27.805 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-26T02:08:27.805 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:27.805 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:27.806 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-26 02:08:20.277819819 +0000 2024-03-26T02:08:27.806 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-26 02:05:10.774141443 +0000 2024-03-26T02:08:27.806 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-26 02:05:10.774141443 +0000 2024-03-26T02:08:27.806 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-26 02:05:10.763141636 +0000 2024-03-26T02:08:27.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-26T02:08:27.872 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-26T02:08:27.873 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-26T02:08:27.873 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00026687 s, 1.9 MB/s 2024-03-26T02:08:27.875 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-26T02:08:27.931 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/nvme4n1 2024-03-26T02:08:27.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:27 smithi165 ceph-mon[27627]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:27.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:27 smithi052 ceph-mon[25889]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:27.986 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/nvme4n1 2024-03-26T02:08:27.987 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-26T02:08:27.987 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-26T02:08:27.987 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-26T02:08:27.987 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-26T02:08:27.987 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-26 02:08:20.289819609 +0000 2024-03-26T02:08:27.987 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-26 02:05:10.924138807 +0000 2024-03-26T02:08:27.987 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-26 02:05:10.924138807 +0000 2024-03-26T02:08:27.988 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-26 02:05:10.915138965 +0000 2024-03-26T02:08:27.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-26T02:08:28.053 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-26T02:08:28.054 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-26T02:08:28.054 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000197482 s, 2.6 MB/s 2024-03-26T02:08:28.055 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-26T02:08:28.111 INFO:tasks.cephadm:Deploying osd.0 on smithi052 with /dev/nvme4n1... 2024-03-26T02:08:28.111 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T02:08:28.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:27 smithi191 podman[30264]: 2024-03-26 02:08:27.892520361 +0000 UTC m=+0.345791381 container init 1267993bed80a6a8833a2fd7ff26adb694c5ec954286fe7ee064074a6859446a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-26T02:08:28.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:27 smithi191 podman[30264]: 2024-03-26 02:08:27.897941328 +0000 UTC m=+0.351212348 container start 1267993bed80a6a8833a2fd7ff26adb694c5ec954286fe7ee064074a6859446a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T02:08:28.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:27 smithi191 ceph-mgr[30409]: -- 172.21.15.191:0/2664264406 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561f5fd3e1a0 con 0x561f5fd14800 2024-03-26T02:08:28.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:28 smithi191 bash[30264]: 1267993bed80a6a8833a2fd7ff26adb694c5ec954286fe7ee064074a6859446a 2024-03-26T02:08:28.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:28 smithi191 systemd[1]: Started Ceph mgr.z for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:08:28.368 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:08:28.787 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:28.449+0000 7f8489c33200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:08:28.787 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:28.542+0000 7f8489c33200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:08:29.354 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.354 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.354 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.354 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:08:29.354 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:08:29.354 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:08:29.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:29.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: Deploying daemon mgr.x on smithi165 2024-03-26T02:08:29.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.355 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:29 smithi052 ceph-mon[25889]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:29.407 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:29.175+0000 7f8489c33200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:08:29.408 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:29.265+0000 7f8489c33200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:08:29.408 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:29.406+0000 7f8489c33200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:08:29.408 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.408 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.408 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.408 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:08:29.408 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:08:29.408 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:08:29.409 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:29.409 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: Deploying daemon mgr.x on smithi165 2024-03-26T02:08:29.409 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.409 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.409 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.409 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-mon[27701]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:29.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:08:29.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:08:29.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:08:29.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:29.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: Deploying daemon mgr.x on smithi165 2024-03-26T02:08:29.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:29.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:29 smithi165 ceph-mon[27627]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:29.787 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:29.501+0000 7f8489c33200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:08:29.787 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:29.583+0000 7f8489c33200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:08:30.128 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:29.843+0000 7f8489c33200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:08:30.128 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:29 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:29.951+0000 7f8489c33200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:08:30.128 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:30 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:30.035+0000 7f8489c33200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:08:30.128 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:30 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:30.127+0000 7f8489c33200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:08:30.178 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T02:08:30.179 INFO:teuthology.orchestra.run.smithi052.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T02:08:30.179 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T02:08:30.179 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-03-26T02:08:30.179 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-03-26T02:08:30.179 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0102942 s, 1.0 GB/s 2024-03-26T02:08:30.179 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-03-26T02:08:30.300 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:30 smithi165 systemd[1]: Starting Ceph mgr.x for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:08:30.537 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:30 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:30.302+0000 7f8489c33200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:08:30.566 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:30 smithi165 podman[29307]: 2024-03-26 02:08:30.202338943 +0000 UTC m=+0.018839807 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:08:30.566 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:30 smithi165 podman[29307]: 2024-03-26 02:08:30.299036026 +0000 UTC m=+0.115536877 container create 8af85cf1aead122b797ab38638d1ab7466e5fc082b32c5736f7b8438b94066bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T02:08:30.566 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:30 smithi165 podman[29307]: 2024-03-26 02:08:30.535183436 +0000 UTC m=+0.351684284 container init 8af85cf1aead122b797ab38638d1ab7466e5fc082b32c5736f7b8438b94066bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.build-date=20240319) 2024-03-26T02:08:30.566 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:30 smithi165 podman[29307]: 2024-03-26 02:08:30.540907332 +0000 UTC m=+0.357408182 container start 8af85cf1aead122b797ab38638d1ab7466e5fc082b32c5736f7b8438b94066bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:08:30.566 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:30 smithi165 ceph-mgr[29353]: -- 172.21.15.165:0/1275586501 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55940fa6a1a0 con 0x55940fa40800 2024-03-26T02:08:30.721 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi052:/dev/nvme4n1 2024-03-26T02:08:30.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:30 smithi165 ceph-mon[27627]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:30.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:30 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:30.952 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:30 smithi165 bash[29307]: 8af85cf1aead122b797ab38638d1ab7466e5fc082b32c5736f7b8438b94066bd 2024-03-26T02:08:30.952 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:30 smithi165 systemd[1]: Started Ceph mgr.x for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:08:30.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:30 smithi052 ceph-mon[25889]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:30.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:30 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:30.983 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:08:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:30 smithi191 ceph-mon[27701]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:31.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:30 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:31.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:31.128+0000 7f98a3dab200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:08:31.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:31.244+0000 7f98a3dab200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:08:31.520 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:31.294+0000 7f8489c33200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:08:31.520 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:31.428+0000 7f8489c33200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:08:31.787 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:31.769+0000 7f8489c33200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:08:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:31.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:31.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:31.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:08:31.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T02:08:31.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:08:31.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:08:31.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:31.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-mon[27701]: Reconfiguring daemon mgr.y on smithi052 2024-03-26T02:08:32.019 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:31.915+0000 7f98a3dab200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:08:32.019 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:32 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:32.018+0000 7f98a3dab200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:08:32.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:32.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:32.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:32.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:32.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:32.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:32.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:32.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:08:32.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T02:08:32.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:08:32.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:08:32.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:32.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:31 smithi165 ceph-mon[27627]: Reconfiguring daemon mgr.y on smithi052 2024-03-26T02:08:32.042 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:32.042 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:32.042 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:32.042 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:32.042 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:32.043 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:32.043 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:32.043 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:08:32.043 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T02:08:32.043 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:08:32.043 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:08:32.043 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:32.043 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:31 smithi052 ceph-mon[25889]: Reconfiguring daemon mgr.y on smithi052 2024-03-26T02:08:32.247 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:31 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:31.861+0000 7f8489c33200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:08:32.247 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:32 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:32.062+0000 7f8489c33200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:08:32.247 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:32 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:32.246+0000 7f8489c33200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:08:32.283 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:32 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:32.161+0000 7f98a3dab200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:08:32.283 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:32 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:32.258+0000 7f98a3dab200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:08:32.620 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:32 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:32.368+0000 7f98a3dab200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:08:32.856 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:32 smithi191 ceph-mon[27701]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:32.856 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:32 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:32.559+0000 7f8489c33200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:08:32.856 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:32 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:32.854+0000 7f8489c33200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:08:32.883 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:32 smithi165 ceph-mon[27627]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:32.883 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:32 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:32.657+0000 7f98a3dab200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:08:32.883 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:32 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:32.771+0000 7f98a3dab200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:08:32.883 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:32 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:32.882+0000 7f98a3dab200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:08:32.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:32 smithi052 ceph-mon[25889]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:33.120 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:33 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:33.103+0000 7f8489c33200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:08:33.164 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:33 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:33.000+0000 7f98a3dab200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:08:33.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:33 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:33.192+0000 7f98a3dab200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:08:34.174 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:33 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:33.952+0000 7f8489c33200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:08:34.363 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:34.235+0000 7f98a3dab200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:08:34.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:08:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:34.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:34.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:34.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:34.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:08:34.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:08:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:08:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.606 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.606 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.606 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:34.606 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:34.606 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:34.606 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:08:34.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:08:34.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:08:34.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:34.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.607 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:34.702 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:34.386+0000 7f98a3dab200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:08:34.787 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:34.535+0000 7f8489c33200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:08:34.787 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:34.630+0000 7f8489c33200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:08:35.011 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:34.725+0000 7f98a3dab200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:08:35.012 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:34 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:34.815+0000 7f98a3dab200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:08:35.012 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:35.011+0000 7f98a3dab200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:08:35.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:08:34 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:08:34.843+0000 7f8489c33200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:08:35.352 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:35.193+0000 7f98a3dab200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:08:35.589 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:35.589 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.589 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.589 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.589 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:35.589 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:35.589 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:35.589 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.589 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:35.589 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.590 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: Standby manager daemon z started 2024-03-26T02:08:35.590 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T02:08:35.590 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:08:35.590 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T02:08:35.590 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:35 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:08:35.702 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:35.474+0000 7f98a3dab200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:08:35.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:35.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:35.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:35.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:35.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:35.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: Standby manager daemon z started 2024-03-26T02:08:35.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T02:08:35.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:08:35.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T02:08:35.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:08:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:35.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: Standby manager daemon z started 2024-03-26T02:08:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T02:08:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:08:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T02:08:35.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:35 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.191:0/3525250864' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:08:36.202 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:35.736+0000 7f98a3dab200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:08:36.202 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:35 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:35.974+0000 7f98a3dab200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:08:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:36 smithi191 ceph-mon[27701]: mgrmap e14: y(active, since 81s), standbys: z 2024-03-26T02:08:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:36 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T02:08:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:36 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3860781476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67c1a134-0690-420c-b67f-fb0239b4361f"}]: dispatch 2024-03-26T02:08:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:36 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3860781476' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67c1a134-0690-420c-b67f-fb0239b4361f"}]': finished 2024-03-26T02:08:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:36 smithi191 ceph-mon[27701]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T02:08:37.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:36 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:37.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:36 smithi191 ceph-mon[27701]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:37.202 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:36 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:36.781+0000 7f98a3dab200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:08:37.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:36 smithi165 ceph-mon[27627]: mgrmap e14: y(active, since 81s), standbys: z 2024-03-26T02:08:37.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:36 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T02:08:37.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:36 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3860781476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67c1a134-0690-420c-b67f-fb0239b4361f"}]: dispatch 2024-03-26T02:08:37.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:36 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3860781476' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67c1a134-0690-420c-b67f-fb0239b4361f"}]': finished 2024-03-26T02:08:37.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:36 smithi165 ceph-mon[27627]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T02:08:37.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:36 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:37.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:36 smithi165 ceph-mon[27627]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:37.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:36 smithi052 ceph-mon[25889]: mgrmap e14: y(active, since 81s), standbys: z 2024-03-26T02:08:37.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:36 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T02:08:37.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:36 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3860781476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67c1a134-0690-420c-b67f-fb0239b4361f"}]: dispatch 2024-03-26T02:08:37.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:36 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3860781476' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67c1a134-0690-420c-b67f-fb0239b4361f"}]': finished 2024-03-26T02:08:37.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:36 smithi052 ceph-mon[25889]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T02:08:37.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:36 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:37.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:36 smithi052 ceph-mon[25889]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:37.661 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:37 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:37.349+0000 7f98a3dab200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:08:37.661 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:37 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:37.446+0000 7f98a3dab200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:08:37.661 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:08:37 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:08:37.660+0000 7f98a3dab200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:08:38.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:38 smithi052 ceph-mon[25889]: Standby manager daemon x started 2024-03-26T02:08:38.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:38 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:08:38.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:38 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:08:38.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:38 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:08:38.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:38 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:08:38.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:38 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/98542442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:08:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:38 smithi191 ceph-mon[27701]: Standby manager daemon x started 2024-03-26T02:08:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:38 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:08:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:38 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:08:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:38 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:08:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:38 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:08:38.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:38 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/98542442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:08:38.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:38 smithi165 ceph-mon[27627]: Standby manager daemon x started 2024-03-26T02:08:38.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:38 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:08:38.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:38 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:08:38.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:38 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:08:38.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:38 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.165:0/3215353327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:08:38.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:38 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/98542442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:08:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:39 smithi191 ceph-mon[27701]: mgrmap e15: y(active, since 83s), standbys: z, x 2024-03-26T02:08:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:39 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:08:39.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:39 smithi191 ceph-mon[27701]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:39.923 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:39 smithi052 ceph-mon[25889]: mgrmap e15: y(active, since 83s), standbys: z, x 2024-03-26T02:08:39.924 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:39 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:08:39.924 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:39 smithi052 ceph-mon[25889]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:39.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:39 smithi165 ceph-mon[27627]: mgrmap e15: y(active, since 83s), standbys: z, x 2024-03-26T02:08:39.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:39 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:08:39.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:39 smithi165 ceph-mon[27627]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:41.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:40 smithi191 ceph-mon[27701]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:41.119 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:40 smithi052 ceph-mon[25889]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:41.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:40 smithi165 ceph-mon[27627]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:43.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:42 smithi191 ceph-mon[27701]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:43.165 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:42 smithi052 ceph-mon[25889]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:43.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:42 smithi165 ceph-mon[27627]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:44.028 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:08:44.028 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:44.028 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:43 smithi052 ceph-mon[25889]: Deploying daemon osd.0 on smithi052 2024-03-26T02:08:44.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:43 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:08:44.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:43 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:44.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:43 smithi191 ceph-mon[27701]: Deploying daemon osd.0 on smithi052 2024-03-26T02:08:44.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:43 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:08:44.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:43 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:44.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:43 smithi165 ceph-mon[27627]: Deploying daemon osd.0 on smithi052 2024-03-26T02:08:44.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:44 smithi052 ceph-mon[25889]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:45.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:44 smithi191 ceph-mon[27701]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:45.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:44 smithi165 ceph-mon[27627]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:47.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:46 smithi191 ceph-mon[27701]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:47.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:46 smithi052 ceph-mon[25889]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:47.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:46 smithi165 ceph-mon[27627]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:49.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:49.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:49.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:49.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:49.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:49.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:49.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: Reconfiguring osd.0 (unknown last config time)... 2024-03-26T02:08:49.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:08:49.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:49.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:49 smithi165 ceph-mon[27627]: Reconfiguring daemon osd.0 on smithi052 2024-03-26T02:08:49.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: Reconfiguring osd.0 (unknown last config time)... 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:49.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:49 smithi052 ceph-mon[25889]: Reconfiguring daemon osd.0 on smithi052 2024-03-26T02:08:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:49.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: Reconfiguring osd.0 (unknown last config time)... 2024-03-26T02:08:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:08:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:49.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:49 smithi191 ceph-mon[27701]: Reconfiguring daemon osd.0 on smithi052 2024-03-26T02:08:50.584 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:50 smithi052 ceph-mon[25889]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:08:50.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:50.585 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:50.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:50 smithi165 ceph-mon[27627]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:08:50.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:50 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:50.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:50 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:50.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:50 smithi191 ceph-mon[27701]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:08:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:50 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:50 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:51.566 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:51 smithi191 ceph-mon[27701]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:51.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:51 smithi165 ceph-mon[27627]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:51.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:51 smithi052 ceph-mon[25889]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:53.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:53.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:08:53.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:08:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:53.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:53.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:54.548 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 0 on host 'smithi052' 2024-03-26T02:08:54.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:54 smithi165 ceph-mon[27627]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:54.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:54 smithi165 ceph-mon[27627]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:08:54.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:54 smithi165 ceph-mon[27627]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T02:08:54.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:54 smithi165 ceph-mon[27627]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-03-26T02:08:54.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:54 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:54.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:54 smithi165 ceph-mon[27627]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:08:54.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:54 smithi165 ceph-mon[27627]: Cluster is now healthy 2024-03-26T02:08:54.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:54 smithi165 ceph-mon[27627]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-03-26T02:08:54.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:54 smithi165 ceph-mon[27627]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T02:08:54.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:54 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:54 smithi191 ceph-mon[27701]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:54 smithi191 ceph-mon[27701]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:08:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:54 smithi191 ceph-mon[27701]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T02:08:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:54 smithi191 ceph-mon[27701]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-03-26T02:08:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:54 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:54 smithi191 ceph-mon[27701]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:08:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:54 smithi191 ceph-mon[27701]: Cluster is now healthy 2024-03-26T02:08:54.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:54 smithi191 ceph-mon[27701]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-03-26T02:08:54.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:54 smithi191 ceph-mon[27701]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T02:08:54.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:54 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:54.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:54 smithi052 ceph-mon[25889]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:54.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:54 smithi052 ceph-mon[25889]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:08:54.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:54 smithi052 ceph-mon[25889]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T02:08:54.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:54 smithi052 ceph-mon[25889]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-03-26T02:08:54.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:54 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:54.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:54 smithi052 ceph-mon[25889]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:08:54.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:54 smithi052 ceph-mon[25889]: Cluster is now healthy 2024-03-26T02:08:54.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:54 smithi052 ceph-mon[25889]: from='osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-03-26T02:08:54.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:54 smithi052 ceph-mon[25889]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T02:08:54.963 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:54 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:55.295 DEBUG:teuthology.orchestra.run.smithi052:osd.0> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T02:08:55.298 INFO:tasks.cephadm:Deploying osd.1 on smithi165 with /dev/nvme4n1... 2024-03-26T02:08:55.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T02:08:55.546 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:08:55.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:55.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:55.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:08:55.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:55 smithi165 ceph-mon[27627]: Adjusting osd_memory_target on smithi052 to 16069M 2024-03-26T02:08:55.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:55.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:55.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.670 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:08:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:55 smithi052 ceph-mon[25889]: Adjusting osd_memory_target on smithi052 to 16069M 2024-03-26T02:08:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:08:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:55 smithi191 ceph-mon[27701]: Adjusting osd_memory_target on smithi052 to 16069M 2024-03-26T02:08:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:55.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:55.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:08:55.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:56.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:56 smithi165 ceph-mon[27627]: purged_snaps scrub starts 2024-03-26T02:08:56.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:56 smithi165 ceph-mon[27627]: purged_snaps scrub ok 2024-03-26T02:08:56.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:56 smithi165 ceph-mon[27627]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:56.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:56 smithi165 ceph-mon[27627]: osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251] boot 2024-03-26T02:08:56.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:56 smithi165 ceph-mon[27627]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T02:08:56.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:56 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:56 smithi191 ceph-mon[27701]: purged_snaps scrub starts 2024-03-26T02:08:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:56 smithi191 ceph-mon[27701]: purged_snaps scrub ok 2024-03-26T02:08:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:56 smithi191 ceph-mon[27701]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:56 smithi191 ceph-mon[27701]: osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251] boot 2024-03-26T02:08:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:56 smithi191 ceph-mon[27701]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T02:08:56.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:56 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:56.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:56 smithi052 ceph-mon[25889]: purged_snaps scrub starts 2024-03-26T02:08:56.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:56 smithi052 ceph-mon[25889]: purged_snaps scrub ok 2024-03-26T02:08:56.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:56 smithi052 ceph-mon[25889]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:08:56.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:56 smithi052 ceph-mon[25889]: osd.0 [v2:172.21.15.52:6802/781166251,v1:172.21.15.52:6803/781166251] boot 2024-03-26T02:08:56.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:56 smithi052 ceph-mon[25889]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T02:08:56.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:56 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:08:57.431 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T02:08:57.431 INFO:teuthology.orchestra.run.smithi165.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T02:08:57.431 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T02:08:57.431 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-03-26T02:08:57.431 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-03-26T02:08:57.431 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126143 s, 831 MB/s 2024-03-26T02:08:57.431 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-03-26T02:08:57.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:57 smithi165 ceph-mon[27627]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T02:08:57.834 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:57 smithi052 ceph-mon[25889]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T02:08:58.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:57 smithi191 ceph-mon[27701]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T02:08:58.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi165:/dev/nvme4n1 2024-03-26T02:08:58.320 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:08:58.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:08:58 smithi165 ceph-mon[27627]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:08:58.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:08:58 smithi052 ceph-mon[25889]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:08:59.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:08:58 smithi191 ceph-mon[27701]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:00.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:00 smithi165 ceph-mon[27627]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:00.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:00 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:09:00.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:00 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:09:00.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:00 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:00.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:00 smithi052 ceph-mon[25889]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:00.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:00 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:09:00.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:00 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:09:00.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:00 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:00 smithi191 ceph-mon[27701]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:00 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:09:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:00 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:09:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:00 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:01.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:01 smithi165 ceph-mon[27627]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:01.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:01 smithi052 ceph-mon[25889]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:01 smithi191 ceph-mon[27701]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:02.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:02 smithi165 ceph-mon[27627]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:02.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:02 smithi052 ceph-mon[25889]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:03.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:02 smithi191 ceph-mon[27701]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:03.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:03 smithi165 ceph-mon[27627]: from='client.? 172.21.15.165:0/1362852692' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ecdb53-2378-496c-adf5-2fb53c8ddc94"}]: dispatch 2024-03-26T02:09:03.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:03 smithi165 ceph-mon[27627]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ecdb53-2378-496c-adf5-2fb53c8ddc94"}]: dispatch 2024-03-26T02:09:03.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:03 smithi165 ceph-mon[27627]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9ecdb53-2378-496c-adf5-2fb53c8ddc94"}]': finished 2024-03-26T02:09:03.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:03 smithi165 ceph-mon[27627]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T02:09:03.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:03 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:03.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:03 smithi052 ceph-mon[25889]: from='client.? 172.21.15.165:0/1362852692' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ecdb53-2378-496c-adf5-2fb53c8ddc94"}]: dispatch 2024-03-26T02:09:03.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:03 smithi052 ceph-mon[25889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ecdb53-2378-496c-adf5-2fb53c8ddc94"}]: dispatch 2024-03-26T02:09:03.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:03 smithi052 ceph-mon[25889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9ecdb53-2378-496c-adf5-2fb53c8ddc94"}]': finished 2024-03-26T02:09:03.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:03 smithi052 ceph-mon[25889]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T02:09:03.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:03 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:03 smithi191 ceph-mon[27701]: from='client.? 172.21.15.165:0/1362852692' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ecdb53-2378-496c-adf5-2fb53c8ddc94"}]: dispatch 2024-03-26T02:09:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:03 smithi191 ceph-mon[27701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ecdb53-2378-496c-adf5-2fb53c8ddc94"}]: dispatch 2024-03-26T02:09:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:03 smithi191 ceph-mon[27701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9ecdb53-2378-496c-adf5-2fb53c8ddc94"}]': finished 2024-03-26T02:09:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:03 smithi191 ceph-mon[27701]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T02:09:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:03 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:04.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:04 smithi165 ceph-mon[27627]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:04.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:04 smithi165 ceph-mon[27627]: from='client.? 172.21.15.165:0/2567527760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:09:04.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:04 smithi052 ceph-mon[25889]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:04.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:04 smithi052 ceph-mon[25889]: from='client.? 172.21.15.165:0/2567527760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:09:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:04 smithi191 ceph-mon[27701]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:05.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:04 smithi191 ceph-mon[27701]: from='client.? 172.21.15.165:0/2567527760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:09:06.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:06 smithi165 ceph-mon[27627]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:06.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:06 smithi191 ceph-mon[27701]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:06.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:06 smithi052 ceph-mon[25889]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:08.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:08 smithi052 ceph-mon[25889]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:08.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:08.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:08.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:08 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:08.843 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:08 smithi165 ceph-mon[27627]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:08.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:08 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:08.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:08 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:08.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:08 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:08 smithi191 ceph-mon[27701]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:08 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:09.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:09 smithi052 ceph-mon[25889]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:09.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:09 smithi052 ceph-mon[25889]: Detected new or changed devices on smithi052 2024-03-26T02:09:09.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:09 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:09.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:09 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:09.905 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:09 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:09.906 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:09 smithi052 ceph-mon[25889]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:09.906 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:09 smithi052 ceph-mon[25889]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:09.906 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:09 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:09.906 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:09 smithi052 ceph-mon[25889]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:09:09.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:09 smithi165 ceph-mon[27627]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:09.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:09 smithi165 ceph-mon[27627]: Detected new or changed devices on smithi052 2024-03-26T02:09:09.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:09 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:09.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:09 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:09.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:09 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:09.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:09 smithi165 ceph-mon[27627]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:09.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:09 smithi165 ceph-mon[27627]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:09.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:09 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:09.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:09 smithi165 ceph-mon[27627]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:09:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:09 smithi191 ceph-mon[27701]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:09 smithi191 ceph-mon[27701]: Detected new or changed devices on smithi052 2024-03-26T02:09:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:09 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:09 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:09 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:09 smithi191 ceph-mon[27701]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:09 smithi191 ceph-mon[27701]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:09 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:10.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:09 smithi191 ceph-mon[27701]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:09:11.196 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:10 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.196 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:10 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.196 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:10 smithi165 ceph-mon[27627]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:11.196 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:10 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.196 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:10 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:11.196 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:10 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:11.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:10 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:11.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:10 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:10 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:09:11.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:10 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:11.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:10 smithi165 ceph-mon[27627]: Deploying daemon osd.1 on smithi165 2024-03-26T02:09:11.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:10 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:10 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:10 smithi052 ceph-mon[25889]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:11.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:10 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:10 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:11.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:10 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:11.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:10 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:11.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:10 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:10 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:09:11.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:10 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:11.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:10 smithi052 ceph-mon[25889]: Deploying daemon osd.1 on smithi165 2024-03-26T02:09:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:10 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:10 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:10 smithi191 ceph-mon[27701]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:11.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:10 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:10 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:11.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:10 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:11.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:10 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:11.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:10 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:11.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:10 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:09:11.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:10 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:11.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:10 smithi191 ceph-mon[27701]: Deploying daemon osd.1 on smithi165 2024-03-26T02:09:12.132 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:11 smithi165 ceph-mon[27627]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:09:12.132 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:11 smithi165 ceph-mon[27627]: Cluster is now healthy 2024-03-26T02:09:12.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:11 smithi191 ceph-mon[27701]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:09:12.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:11 smithi191 ceph-mon[27701]: Cluster is now healthy 2024-03-26T02:09:12.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:11 smithi052 ceph-mon[25889]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:09:12.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:11 smithi052 ceph-mon[25889]: Cluster is now healthy 2024-03-26T02:09:13.054 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:12 smithi165 ceph-mon[27627]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:13.102 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:12 smithi191 ceph-mon[27701]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:13.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:12 smithi052 ceph-mon[25889]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:14.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:14 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.343 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:14 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.344 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:14 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.344 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:14 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.344 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:14 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.344 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:14 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:14 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:14 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:14 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:14 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:14 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.349 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:14 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:14.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:14 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:15.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:15 smithi191 ceph-mon[27701]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:15.896 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:15 smithi165 ceph-mon[27627]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:15.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:15 smithi052 ceph-mon[25889]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:17.542 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:17.542 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.542 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.542 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:17.542 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:17.542 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:17.542 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:17.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:17.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:17.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:17.543 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:17 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.762 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:17.762 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.762 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.762 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:17.762 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:17.762 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:17.763 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.763 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.763 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.763 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:17.763 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.763 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:17.763 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:17.763 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:17.763 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:17 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:17.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:17.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:17.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:17.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:17.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:17 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:18.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:18 smithi052 ceph-mon[25889]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:18.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:18 smithi052 ceph-mon[25889]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:18.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:18 smithi052 ceph-mon[25889]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:09:18.786 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:18 smithi191 ceph-mon[27701]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:18 smithi191 ceph-mon[27701]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:18.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:18 smithi191 ceph-mon[27701]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:09:18.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:18 smithi165 ceph-mon[27627]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:18.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:18 smithi165 ceph-mon[27627]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:18.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:18 smithi165 ceph-mon[27627]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:09:20.362 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 1 on host 'smithi165' 2024-03-26T02:09:20.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:20 smithi165 ceph-mon[27627]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:20.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:20 smithi165 ceph-mon[27627]: from='osd.1 [v2:172.21.15.165:6800/3456552919,v1:172.21.15.165:6801/3456552919]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:09:20.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:20 smithi165 ceph-mon[27627]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:09:20.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:20.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:20.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:09:20.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:20.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:20.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:20.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:20.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:20 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:20.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:20 smithi191 ceph-mon[27701]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:20 smithi191 ceph-mon[27701]: from='osd.1 [v2:172.21.15.165:6800/3456552919,v1:172.21.15.165:6801/3456552919]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:09:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:20 smithi191 ceph-mon[27701]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:09:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:09:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:20.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:20 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:20 smithi052 ceph-mon[25889]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:20 smithi052 ceph-mon[25889]: from='osd.1 [v2:172.21.15.165:6800/3456552919,v1:172.21.15.165:6801/3456552919]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:09:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:20 smithi052 ceph-mon[25889]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:09:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:09:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:20.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:20.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:20.962 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:20 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:21.068 DEBUG:teuthology.orchestra.run.smithi165:osd.1> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T02:09:21.071 INFO:tasks.cephadm:Deploying osd.2 on smithi191 with /dev/nvme4n1... 2024-03-26T02:09:21.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-26T02:09:21.316 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:09:21.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:21 smithi191 ceph-mon[27701]: Adjusting osd_memory_target on smithi165 to 16064M 2024-03-26T02:09:21.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:21 smithi191 ceph-mon[27701]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:21.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:21 smithi191 ceph-mon[27701]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:21.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:21 smithi191 ceph-mon[27701]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:09:21.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:21 smithi191 ceph-mon[27701]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T02:09:21.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:21 smithi191 ceph-mon[27701]: from='osd.1 [v2:172.21.15.165:6800/3456552919,v1:172.21.15.165:6801/3456552919]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-26T02:09:21.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:21 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:21.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:21 smithi191 ceph-mon[27701]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-26T02:09:21.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:21 smithi165 ceph-mon[27627]: Adjusting osd_memory_target on smithi165 to 16064M 2024-03-26T02:09:21.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:21 smithi165 ceph-mon[27627]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:21.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:21 smithi165 ceph-mon[27627]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:21.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:21 smithi165 ceph-mon[27627]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:09:21.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:21 smithi165 ceph-mon[27627]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T02:09:21.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:21 smithi165 ceph-mon[27627]: from='osd.1 [v2:172.21.15.165:6800/3456552919,v1:172.21.15.165:6801/3456552919]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-26T02:09:21.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:21 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:21.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:21 smithi165 ceph-mon[27627]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-26T02:09:21.953 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:09:21 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1[33193]: 2024-03-26T02:09:21.504+0000 7f5ddb977700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T02:09:21.953 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:09:21 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1[33193]: 2024-03-26T02:09:21.507+0000 7f5dd776a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:09:21.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:21 smithi052 ceph-mon[25889]: Adjusting osd_memory_target on smithi165 to 16064M 2024-03-26T02:09:21.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:21 smithi052 ceph-mon[25889]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:21.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:21 smithi052 ceph-mon[25889]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:21.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:21 smithi052 ceph-mon[25889]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:09:21.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:21 smithi052 ceph-mon[25889]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T02:09:21.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:21 smithi052 ceph-mon[25889]: from='osd.1 [v2:172.21.15.165:6800/3456552919,v1:172.21.15.165:6801/3456552919]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-26T02:09:21.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:21 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:21.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:21 smithi052 ceph-mon[25889]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-26T02:09:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:22 smithi191 ceph-mon[27701]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-26T02:09:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:22 smithi191 ceph-mon[27701]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T02:09:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:22 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:22.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:22 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:22.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:22 smithi165 ceph-mon[27627]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-26T02:09:22.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:22 smithi165 ceph-mon[27627]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T02:09:22.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:22 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:22.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:22 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:22 smithi052 ceph-mon[25889]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-26T02:09:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:22 smithi052 ceph-mon[25889]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T02:09:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:22 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:22 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:23.416 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/nvme4n1 2024-03-26T02:09:23.416 INFO:teuthology.orchestra.run.smithi191.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-26T02:09:23.416 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-26T02:09:23.417 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-03-26T02:09:23.417 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-03-26T02:09:23.417 INFO:teuthology.orchestra.run.smithi191.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124222 s, 844 MB/s 2024-03-26T02:09:23.417 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-03-26T02:09:23.720 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:23 smithi191 ceph-mon[27701]: purged_snaps scrub starts 2024-03-26T02:09:23.721 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:23 smithi191 ceph-mon[27701]: purged_snaps scrub ok 2024-03-26T02:09:23.721 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:23 smithi191 ceph-mon[27701]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:23.721 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:23 smithi191 ceph-mon[27701]: osd.1 [v2:172.21.15.165:6800/3456552919,v1:172.21.15.165:6801/3456552919] boot 2024-03-26T02:09:23.721 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:23 smithi191 ceph-mon[27701]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T02:09:23.721 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:23 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:23.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:23 smithi165 ceph-mon[27627]: purged_snaps scrub starts 2024-03-26T02:09:23.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:23 smithi165 ceph-mon[27627]: purged_snaps scrub ok 2024-03-26T02:09:23.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:23 smithi165 ceph-mon[27627]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:23.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:23 smithi165 ceph-mon[27627]: osd.1 [v2:172.21.15.165:6800/3456552919,v1:172.21.15.165:6801/3456552919] boot 2024-03-26T02:09:23.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:23 smithi165 ceph-mon[27627]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T02:09:23.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:23 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:23.955 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi191:/dev/nvme4n1 2024-03-26T02:09:23.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:23 smithi052 ceph-mon[25889]: purged_snaps scrub starts 2024-03-26T02:09:23.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:23 smithi052 ceph-mon[25889]: purged_snaps scrub ok 2024-03-26T02:09:23.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:23 smithi052 ceph-mon[25889]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:09:23.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:23 smithi052 ceph-mon[25889]: osd.1 [v2:172.21.15.165:6800/3456552919,v1:172.21.15.165:6801/3456552919] boot 2024-03-26T02:09:23.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:23 smithi052 ceph-mon[25889]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T02:09:23.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:23 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:09:24.186 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:09:24.582 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:24 smithi191 ceph-mon[27701]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T02:09:24.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:24 smithi165 ceph-mon[27627]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T02:09:24.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:24 smithi052 ceph-mon[25889]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T02:09:25.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:25 smithi191 ceph-mon[27701]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:25.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:25 smithi165 ceph-mon[27627]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:25.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:25 smithi052 ceph-mon[25889]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:26 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:09:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:26 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:09:26.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:26 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:26.805 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:26 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:09:26.805 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:26 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:09:26.805 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:26 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:26.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:26 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:09:26.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:26 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:09:26.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:26 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:27 smithi191 ceph-mon[27701]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:27.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:27 smithi191 ceph-mon[27701]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:27.877 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:27 smithi052 ceph-mon[25889]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:27.877 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:27 smithi052 ceph-mon[25889]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:27.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:27 smithi165 ceph-mon[27627]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:27.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:27 smithi165 ceph-mon[27627]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:29.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:29.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:29.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:29 smithi052 ceph-mon[25889]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:29.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:29 smithi052 ceph-mon[25889]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:29.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:29.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:29 smithi052 ceph-mon[25889]: from='client.? 172.21.15.191:0/2560863575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "896c1548-6eb5-4713-a7f9-39f4e7e6c7f0"}]: dispatch 2024-03-26T02:09:29.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:29 smithi052 ceph-mon[25889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "896c1548-6eb5-4713-a7f9-39f4e7e6c7f0"}]: dispatch 2024-03-26T02:09:29.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:29 smithi052 ceph-mon[25889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "896c1548-6eb5-4713-a7f9-39f4e7e6c7f0"}]': finished 2024-03-26T02:09:29.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:29 smithi052 ceph-mon[25889]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T02:09:29.212 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:29 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:29.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:29 smithi165 ceph-mon[27627]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:29 smithi165 ceph-mon[27627]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:29 smithi165 ceph-mon[27627]: from='client.? 172.21.15.191:0/2560863575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "896c1548-6eb5-4713-a7f9-39f4e7e6c7f0"}]: dispatch 2024-03-26T02:09:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:29 smithi165 ceph-mon[27627]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "896c1548-6eb5-4713-a7f9-39f4e7e6c7f0"}]: dispatch 2024-03-26T02:09:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:29 smithi165 ceph-mon[27627]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "896c1548-6eb5-4713-a7f9-39f4e7e6c7f0"}]': finished 2024-03-26T02:09:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:29 smithi165 ceph-mon[27627]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T02:09:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:29 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:29 smithi191 ceph-mon[27701]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:29 smithi191 ceph-mon[27701]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:29 smithi191 ceph-mon[27701]: from='client.? 172.21.15.191:0/2560863575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "896c1548-6eb5-4713-a7f9-39f4e7e6c7f0"}]: dispatch 2024-03-26T02:09:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:29 smithi191 ceph-mon[27701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "896c1548-6eb5-4713-a7f9-39f4e7e6c7f0"}]: dispatch 2024-03-26T02:09:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:29 smithi191 ceph-mon[27701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "896c1548-6eb5-4713-a7f9-39f4e7e6c7f0"}]': finished 2024-03-26T02:09:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:29 smithi191 ceph-mon[27701]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T02:09:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:29 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:30.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:30 smithi165 ceph-mon[27627]: from='client.? 172.21.15.191:0/1956358668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:09:30.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:30 smithi052 ceph-mon[25889]: from='client.? 172.21.15.191:0/1956358668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:09:30.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:30 smithi191 ceph-mon[27701]: from='client.? 172.21.15.191:0/1956358668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:09:31.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:31 smithi165 ceph-mon[27627]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:31.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:31 smithi052 ceph-mon[25889]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:31.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:31 smithi191 ceph-mon[27701]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:33.266 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:33 smithi191 ceph-mon[27701]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:33.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:33 smithi165 ceph-mon[27627]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:33.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:33 smithi052 ceph-mon[25889]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:34.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:34.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:34.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:34 smithi191 ceph-mon[27701]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:34.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:34 smithi191 ceph-mon[27701]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:34.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:34 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:34.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:34.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:34.522 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:34 smithi165 ceph-mon[27627]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:34.522 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:34 smithi165 ceph-mon[27627]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:34.522 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:34 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:34.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:34.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:34.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:34 smithi052 ceph-mon[25889]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:34.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:34 smithi052 ceph-mon[25889]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:34.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:34 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.525 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:36 smithi191 ceph-mon[27701]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:36.525 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:36 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.525 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:36 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.525 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:36 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.526 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:36 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:36.526 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:36 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:36.526 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:36 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:36.526 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:36 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.681 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:36 smithi165 ceph-mon[27627]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:36.681 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:36 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.681 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:36 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.681 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:36 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.681 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:36 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:36.681 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:36 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:36.681 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:36 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:36.681 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:36 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:36 smithi052 ceph-mon[25889]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:36.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:36 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:36 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:36 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:36.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:36 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:36.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:36 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:36.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:36 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:36.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:36 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:37.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:37 smithi191 ceph-mon[27701]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:37.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:37 smithi191 ceph-mon[27701]: Detected new or changed devices on smithi165 2024-03-26T02:09:37.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:37 smithi191 ceph-mon[27701]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:09:37.387 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:37 smithi191 ceph-mon[27701]: Cluster is now healthy 2024-03-26T02:09:37.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:37 smithi165 ceph-mon[27627]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:37.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:37 smithi165 ceph-mon[27627]: Detected new or changed devices on smithi165 2024-03-26T02:09:37.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:37 smithi165 ceph-mon[27627]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:09:37.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:37 smithi165 ceph-mon[27627]: Cluster is now healthy 2024-03-26T02:09:37.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:37 smithi052 ceph-mon[25889]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:37.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:37 smithi052 ceph-mon[25889]: Detected new or changed devices on smithi165 2024-03-26T02:09:37.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:37 smithi052 ceph-mon[25889]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:09:37.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:37 smithi052 ceph-mon[25889]: Cluster is now healthy 2024-03-26T02:09:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:38 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:09:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:38 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:38.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:38 smithi191 ceph-mon[27701]: Deploying daemon osd.2 on smithi191 2024-03-26T02:09:38.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:38 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:09:38.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:38 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:38.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:38 smithi165 ceph-mon[27627]: Deploying daemon osd.2 on smithi191 2024-03-26T02:09:38.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:38 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:09:38.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:38 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:38.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:38 smithi052 ceph-mon[25889]: Deploying daemon osd.2 on smithi191 2024-03-26T02:09:39.545 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:39 smithi191 ceph-mon[27701]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:39.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:39 smithi165 ceph-mon[27627]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:39.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:39 smithi052 ceph-mon[25889]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:41 smithi191 ceph-mon[27701]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:41.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:41 smithi165 ceph-mon[27627]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:41.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:41 smithi052 ceph-mon[25889]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:43.624 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:43 smithi191 ceph-mon[27701]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:43.624 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:43 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:43.624 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:43 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:43.624 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:43 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:43.624 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:43 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:43.624 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:43 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:43.624 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:43 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:43.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:43 smithi165 ceph-mon[27627]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:43.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:43 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:43.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:43 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:43.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:43 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:43.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:43 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:43.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:43 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:43.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:43 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:43.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:43 smithi052 ceph-mon[25889]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:43.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:43.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:43.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:43.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:43.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:43.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:43 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:45.660 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:45 smithi191 ceph-mon[27701]: pgmap v92: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:45.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:45 smithi165 ceph-mon[27627]: pgmap v92: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:45.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:45 smithi052 ceph-mon[25889]: pgmap v92: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:45.993 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 2 on host 'smithi191' 2024-03-26T02:09:46.607 DEBUG:teuthology.orchestra.run.smithi191:osd.2> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T02:09:46.610 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-26T02:09:46.610 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T02:09:46.936 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:09:47.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: pgmap v93: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:47.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:47.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:09:47.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: Adjusting osd_memory_target on smithi191 to 16064M 2024-03-26T02:09:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: from='osd.2 [v2:172.21.15.191:6800/945058458,v1:172.21.15.191:6801/945058458]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:09:47.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:46 smithi191 ceph-mon[27701]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:09:47.288 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: pgmap v93: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:47.288 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.288 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.288 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:47.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:09:47.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: Adjusting osd_memory_target on smithi191 to 16064M 2024-03-26T02:09:47.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:47.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:47.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: from='osd.2 [v2:172.21.15.191:6800/945058458,v1:172.21.15.191:6801/945058458]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:09:47.289 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:46 smithi052 ceph-mon[25889]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:09:47.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: pgmap v93: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:47.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:47.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:09:47.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: Adjusting osd_memory_target on smithi191 to 16064M 2024-03-26T02:09:47.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:47.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:47.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:47.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: from='osd.2 [v2:172.21.15.191:6800/945058458,v1:172.21.15.191:6801/945058458]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:09:47.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:46 smithi165 ceph-mon[27627]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:09:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:48 smithi052 ceph-mon[25889]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:09:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:48 smithi052 ceph-mon[25889]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T02:09:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:48 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:48 smithi052 ceph-mon[25889]: from='osd.2 [v2:172.21.15.191:6800/945058458,v1:172.21.15.191:6801/945058458]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-26T02:09:48.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:48 smithi052 ceph-mon[25889]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-26T02:09:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:48 smithi191 ceph-mon[27701]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:09:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:48 smithi191 ceph-mon[27701]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T02:09:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:48 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:48 smithi191 ceph-mon[27701]: from='osd.2 [v2:172.21.15.191:6800/945058458,v1:172.21.15.191:6801/945058458]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-26T02:09:48.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:48 smithi191 ceph-mon[27701]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-26T02:09:48.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:48 smithi165 ceph-mon[27627]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:09:48.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:48 smithi165 ceph-mon[27627]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T02:09:48.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:48 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:48.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:48 smithi165 ceph-mon[27627]: from='osd.2 [v2:172.21.15.191:6800/945058458,v1:172.21.15.191:6801/945058458]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-26T02:09:48.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:48 smithi165 ceph-mon[27627]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-26T02:09:48.787 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:09:48 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2[34671]: 2024-03-26T02:09:48.508+0000 7f88a5f67700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T02:09:48.787 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:09:48 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2[34671]: 2024-03-26T02:09:48.511+0000 7f889dd52700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:09:49.245 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:49 smithi052 ceph-mon[25889]: pgmap v95: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:49.245 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:49 smithi052 ceph-mon[25889]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-26T02:09:49.246 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:49 smithi052 ceph-mon[25889]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T02:09:49.246 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:49.246 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:49.246 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:49.246 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:49.246 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:49 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:49.246 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:49 smithi052 ceph-mon[25889]: from='osd.2 ' entity='osd.2' 2024-03-26T02:09:49.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:49 smithi191 ceph-mon[27701]: pgmap v95: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:49.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:49 smithi191 ceph-mon[27701]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-26T02:09:49.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:49 smithi191 ceph-mon[27701]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T02:09:49.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:49.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:49.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:49.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:49.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:49 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:49.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:49 smithi191 ceph-mon[27701]: from='osd.2 ' entity='osd.2' 2024-03-26T02:09:49.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:49 smithi165 ceph-mon[27627]: pgmap v95: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:49.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:49 smithi165 ceph-mon[27627]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-26T02:09:49.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:49 smithi165 ceph-mon[27627]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T02:09:49.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:49.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:49.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:49.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:49.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:49 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:49.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:49 smithi165 ceph-mon[27627]: from='osd.2 ' entity='osd.2' 2024-03-26T02:09:49.707 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:09:50.285 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:50 smithi052 ceph-mon[25889]: purged_snaps scrub starts 2024-03-26T02:09:50.285 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:50 smithi052 ceph-mon[25889]: purged_snaps scrub ok 2024-03-26T02:09:50.285 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:50.286 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:50 smithi052 ceph-mon[25889]: osd.2 [v2:172.21.15.191:6800/945058458,v1:172.21.15.191:6801/945058458] boot 2024-03-26T02:09:50.286 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:50 smithi052 ceph-mon[25889]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T02:09:50.286 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:50 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:50.286 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:50 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3098387728' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:09:50.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:50 smithi191 ceph-mon[27701]: purged_snaps scrub starts 2024-03-26T02:09:50.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:50 smithi191 ceph-mon[27701]: purged_snaps scrub ok 2024-03-26T02:09:50.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:50 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:50.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:50 smithi191 ceph-mon[27701]: osd.2 [v2:172.21.15.191:6800/945058458,v1:172.21.15.191:6801/945058458] boot 2024-03-26T02:09:50.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:50 smithi191 ceph-mon[27701]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T02:09:50.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:50 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:50.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:50 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3098387728' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:09:50.311 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":18,"num_osds":3,"num_up_osds":3,"osd_up_since":1711418989,"num_in_osds":3,"osd_in_since":1711418968,"num_remapped_pgs":0} 2024-03-26T02:09:50.311 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:09:50.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:50 smithi165 ceph-mon[27627]: purged_snaps scrub starts 2024-03-26T02:09:50.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:50 smithi165 ceph-mon[27627]: purged_snaps scrub ok 2024-03-26T02:09:50.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:50 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:50.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:50 smithi165 ceph-mon[27627]: osd.2 [v2:172.21.15.191:6800/945058458,v1:172.21.15.191:6801/945058458] boot 2024-03-26T02:09:50.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:50 smithi165 ceph-mon[27627]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T02:09:50.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:50 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:09:50.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:50 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3098387728' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:09:50.552 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:09:51.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:51 smithi052 ceph-mon[25889]: pgmap v98: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:51.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:51 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:51.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:51 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:51.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:51 smithi052 ceph-mon[25889]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T02:09:51.758 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:51 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:51 smithi191 ceph-mon[27701]: pgmap v98: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:51 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:51 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:51.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:51 smithi191 ceph-mon[27701]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T02:09:51.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:51 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:51.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:51 smithi165 ceph-mon[27627]: pgmap v98: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:09:51.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:51 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:51.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:51 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:51.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:51 smithi165 ceph-mon[27627]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T02:09:51.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:51 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:52.484 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:09:52.484 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":20,"fsid":"46f2b2dc-eb15-11ee-95d0-87774f69a715","created":"2024-03-26T02:06:16.299492+0000","modified":"2024-03-26T02:09:52.030417+0000","last_up_change":"2024-03-26T02:09:49.511019+0000","last_in_change":"2024-03-26T02:09:28.702724+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T02:09:51.026808+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67c1a134-0690-420c-b67f-fb0239b4361f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6803","nonce":781166251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6805","nonce":781166251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6809","nonce":781166251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6807","nonce":781166251}]},"public_addr":"172.21.15.52:6803/781166251","cluster_addr":"172.21.15.52:6805/781166251","heartbeat_back_addr":"172.21.15.52:6809/781166251","heartbeat_front_addr":"172.21.15.52:6807/781166251","state":["exists","up"]},{"osd":1,"uuid":"b9ecdb53-2378-496c-adf5-2fb53c8ddc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6801","nonce":3456552919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6803","nonce":3456552919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6807","nonce":3456552919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6805","nonce":3456552919}]},"public_addr":"172.21.15.165:6801/3456552919","cluster_addr":"172.21.15.165:6803/3456552919","heartbeat_back_addr":"172.21.15.165:6807/3456552919","heartbeat_front_addr":"172.21.15.165:6805/3456552919","state":["exists","up"]},{"osd":2,"uuid":"896c1548-6eb5-4713-a7f9-39f4e7e6c7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6801","nonce":945058458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6803","nonce":945058458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6807","nonce":945058458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6805","nonce":945058458}]},"public_addr":"172.21.15.191:6801/945058458","cluster_addr":"172.21.15.191:6803/945058458","heartbeat_back_addr":"172.21.15.191:6807/945058458","heartbeat_front_addr":"172.21.15.191:6805/945058458","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:08:53.708991+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:09:20.587130+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:09:47.048074+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/3063795633":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/808014555":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/4225488723":"2024-03-27T02:06:47.789796+0000","172.21.15.52:6801/1514360646":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/201621324":"2024-03-27T02:06:47.789796+0000","172.21.15.52:0/3458856682":"2024-03-27T02:06:47.789796+0000","172.21.15.52:6800/1514360646":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/426412327":"2024-03-27T02:07:14.675819+0000","172.21.15.52:6800/1384523876":"2024-03-27T02:06:47.789796+0000","172.21.15.52:6801/1384523876":"2024-03-27T02:06:47.789796+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:09:52.656 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:09:52 smithi052 sudo[41187]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T02:09:52.656 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:09:52 smithi052 sudo[41187]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:52.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:09:52 smithi052 sudo[41187]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:52.657 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:09:52 smithi052 sudo[41187]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:52.960 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:09:52 smithi052 sudo[41206]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T02:09:52.961 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:09:52 smithi052 sudo[41206]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:52.961 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:09:52 smithi052 sudo[41206]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:52.961 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:09:52 smithi052 sudo[41206]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:53.017 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:09:51.026808+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T02:09:53.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T02:09:53.179 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:09:52 smithi165 sudo[34779]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T02:09:53.179 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:09:52 smithi165 sudo[34779]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:53.179 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:09:52 smithi165 sudo[34779]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:53.179 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:09:52 smithi165 sudo[34779]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:53.179 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:53 smithi165 ceph-mon[27627]: pgmap v100: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:53.179 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:53.180 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:53 smithi165 ceph-mon[27627]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T02:09:53.180 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:53 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2665616451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:09:53.180 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:53 smithi165 ceph-mon[27627]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:53.180 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.180 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.180 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.180 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:53 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.257 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:09:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:53 smithi191 ceph-mon[27701]: pgmap v100: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:53 smithi191 ceph-mon[27701]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T02:09:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:53 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2665616451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:09:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:53 smithi191 ceph-mon[27701]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:53 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.378 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 ceph-mon[25889]: pgmap v100: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:53.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:53.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 ceph-mon[25889]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T02:09:53.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2665616451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:09:53.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 ceph-mon[25889]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:53.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:53.452 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:09:53 smithi165 sudo[34782]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T02:09:53.452 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:09:53 smithi165 sudo[34782]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:53.452 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:09:53 smithi165 sudo[34782]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:53.452 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:09:53 smithi165 sudo[34782]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:53.695 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:09:53 smithi191 sudo[35789]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-26T02:09:53.695 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:09:53 smithi191 sudo[35789]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:53.695 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:09:53 smithi191 sudo[35789]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:53.696 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:09:53 smithi191 sudo[35789]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:54.037 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:09:53 smithi191 sudo[35792]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-26T02:09:54.037 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:09:53 smithi191 sudo[35792]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:54.037 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:09:53 smithi191 sudo[35792]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:54.037 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:09:53 smithi191 sudo[35792]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:54.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 sudo[41327]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:09:54.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 sudo[41327]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:54.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 sudo[41327]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:54.243 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:53 smithi052 sudo[41327]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:54.331 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 ceph-mon[27701]: Detected new or changed devices on smithi191 2024-03-26T02:09:54.331 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 ceph-mon[27701]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T02:09:54.331 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:54.331 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:54.331 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 ceph-mon[27701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:09:54.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:54 smithi052 sudo[41339]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:09:54.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:54 smithi052 sudo[41339]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:54.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:54 smithi052 sudo[41339]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:54.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:54 smithi052 sudo[41339]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:54.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:54 smithi052 ceph-mon[25889]: Detected new or changed devices on smithi191 2024-03-26T02:09:54.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:54 smithi052 ceph-mon[25889]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T02:09:54.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:54 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:54.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:54 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:54.569 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:54 smithi052 ceph-mon[25889]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:09:54.640 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 sudo[35848]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:09:54.641 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 sudo[35848]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:54.641 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 sudo[35848]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:54.641 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 sudo[35848]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:54.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:54 smithi165 ceph-mon[27627]: Detected new or changed devices on smithi191 2024-03-26T02:09:54.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:54 smithi165 ceph-mon[27627]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T02:09:54.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:54 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:54.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:54 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:54.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:54 smithi165 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:09:55.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 sudo[35867]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:09:55.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 sudo[35867]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:55.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 sudo[35867]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:55.027 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:54 smithi191 sudo[35867]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:55.123 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2024-03-26T02:09:55.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 sudo[34854]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:09:55.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 sudo[34854]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:55.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 sudo[34854]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:55.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 sudo[34854]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:55.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 sudo[34871]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:09:55.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 sudo[34871]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:09:55.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 sudo[34871]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:09:55.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 sudo[34871]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:09:55.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:55.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:09:55.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:09:55.622 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:09:55.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:09:55.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T02:09:55.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:55.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:09:55.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:09:55.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:09:55.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:09:55.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:09:55.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:55.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:09:55.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:09:55.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:09:55.624 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:55 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3740621141' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:09:55.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:09:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:09:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:09:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:09:55.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:55 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3740621141' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:09:55.738 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi052 2024-03-26T02:09:55.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi052=foo.a' 2024-03-26T02:09:55.766 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:55.766 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:09:55.766 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:09:55.766 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:09:55.766 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:09:55.766 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-26T02:09:55.766 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:55.767 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:09:55.767 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:09:55.767 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:09:55.767 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:09:55.767 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:09:55.767 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:55.767 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:09:55.767 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:09:55.767 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:09:55.767 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:55 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3740621141' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:09:55.984 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:09:56.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:56 smithi165 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:09:56.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:56 smithi165 ceph-mon[27627]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:09:56.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:56 smithi165 ceph-mon[27627]: Cluster is now healthy 2024-03-26T02:09:56.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:56 smithi165 ceph-mon[27627]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:09:56.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:56 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:56.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:56 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:56.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:56 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:56.618 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:56 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:56.629 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:56 smithi191 ceph-mon[27701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:09:56.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:56 smithi191 ceph-mon[27701]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:09:56.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:56 smithi191 ceph-mon[27701]: Cluster is now healthy 2024-03-26T02:09:56.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:56 smithi191 ceph-mon[27701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:09:56.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:56 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:56.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:56 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:56.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:56 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:56.630 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:56 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:56.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:56 smithi052 ceph-mon[25889]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:09:56.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:56 smithi052 ceph-mon[25889]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:09:56.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:56 smithi052 ceph-mon[25889]: Cluster is now healthy 2024-03-26T02:09:56.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:56 smithi052 ceph-mon[25889]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:09:56.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:56 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:56.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:56 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:56.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:56 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:56.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:56 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:57.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:57 smithi165 ceph-mon[27627]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:57.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:57 smithi052 ceph-mon[25889]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:57.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:57 smithi191 ceph-mon[27701]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:58.493 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled rgw.foo.a update... 2024-03-26T02:09:59.097 DEBUG:teuthology.orchestra.run.smithi052:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T02:09:59.100 INFO:tasks.cephadm:Adding prometheus.a on smithi165 2024-03-26T02:09:59.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi165=a' 2024-03-26T02:09:59.369 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:09:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi052=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-26T02:09:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:09:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:09:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:09:59 smithi052 ceph-mon[25889]: Deploying daemon rgw.foo.a on smithi052 2024-03-26T02:09:59.749 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:59.749 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi052=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:59.749 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-26T02:09:59.749 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:59.749 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:59.749 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:59.749 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:59.750 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:59.750 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:09:59.750 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:09:59.750 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:59.750 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:59.750 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:09:59 smithi191 ceph-mon[27701]: Deploying daemon rgw.foo.a on smithi052 2024-03-26T02:09:59.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:09:59.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi052=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:59.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-26T02:09:59.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:09:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:09:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:09:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:09:59 smithi165 ceph-mon[27627]: Deploying daemon rgw.foo.a on smithi052 2024-03-26T02:10:00.529 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:00 smithi052 ceph-mon[25889]: overall HEALTH_OK 2024-03-26T02:10:00.679 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:00 smithi191 ceph-mon[27701]: overall HEALTH_OK 2024-03-26T02:10:00.950 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:10:00 smithi052 systemd[1]: Starting Ceph rgw.foo.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:10:00.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:00 smithi165 ceph-mon[27627]: overall HEALTH_OK 2024-03-26T02:10:01.135 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled prometheus update... 2024-03-26T02:10:01.211 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:10:00 smithi052 podman[41738]: 2024-03-26 02:10:00.852491822 +0000 UTC m=+0.018077918 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:10:01.211 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:10:00 smithi052 podman[41738]: 2024-03-26 02:10:00.9733208 +0000 UTC m=+0.138906931 container create 89a19713c515a773a968546d9a57cfe17a61871c40b3afbfc3f0967c3d78f8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:10:01.512 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:10:01 smithi052 podman[41738]: 2024-03-26 02:10:01.224354076 +0000 UTC m=+0.389940164 container init 89a19713c515a773a968546d9a57cfe17a61871c40b3afbfc3f0967c3d78f8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T02:10:01.513 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:10:01 smithi052 podman[41738]: 2024-03-26 02:10:01.229933361 +0000 UTC m=+0.395519454 container start 89a19713c515a773a968546d9a57cfe17a61871c40b3afbfc3f0967c3d78f8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.build-date=20240319) 2024-03-26T02:10:01.513 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:10:01 smithi052 bash[41738]: 89a19713c515a773a968546d9a57cfe17a61871c40b3afbfc3f0967c3d78f8be 2024-03-26T02:10:01.513 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:10:01 smithi052 systemd[1]: Started Ceph rgw.foo.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:10:01.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:01 smithi191 ceph-mon[27701]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 392 KiB/s wr, 25 op/s 2024-03-26T02:10:01.771 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:01 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:01.797 DEBUG:teuthology.orchestra.run.smithi165:prometheus.a> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T02:10:01.799 INFO:tasks.cephadm:Adding node-exporter.a on smithi052 2024-03-26T02:10:01.800 INFO:tasks.cephadm:Adding node-exporter.b on smithi165 2024-03-26T02:10:01.800 INFO:tasks.cephadm:Adding node-exporter.c on smithi191 2024-03-26T02:10:01.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '3;smithi052=a;smithi165=b;smithi191=c' 2024-03-26T02:10:01.824 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:01 smithi165 ceph-mon[27627]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 392 KiB/s wr, 25 op/s 2024-03-26T02:10:01.824 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:01 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:01.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:01 smithi052 ceph-mon[25889]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 392 KiB/s wr, 25 op/s 2024-03-26T02:10:01.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:01 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.028 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:10:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: Saving service prometheus spec with placement smithi165=a;count:1 2024-03-26T02:10:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-26T02:10:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T02:10:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2188108003' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:10:02.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:02 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/246062740' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:10:02.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:02.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: Saving service prometheus spec with placement smithi165=a;count:1 2024-03-26T02:10:02.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.795 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-26T02:10:02.795 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.795 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.795 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:02.795 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:02.795 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:02.795 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T02:10:02.795 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2188108003' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:10:02.795 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:02 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/246062740' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:10:02.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:02.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: Saving service prometheus spec with placement smithi165=a;count:1 2024-03-26T02:10:02.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-26T02:10:02.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:02.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:02.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:02.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:02.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T02:10:02.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2188108003' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:10:02.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:02 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/246062740' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:10:03.811 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled node-exporter update... 2024-03-26T02:10:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:03 smithi191 ceph-mon[27701]: Checking dashboard <-> RGW credentials 2024-03-26T02:10:04.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:03 smithi191 ceph-mon[27701]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 294 KiB/s wr, 18 op/s 2024-03-26T02:10:04.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:03 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2188108003' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:10:04.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:03 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/246062740' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:10:04.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:03 smithi191 ceph-mon[27701]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T02:10:04.194 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:03 smithi052 ceph-mon[25889]: Checking dashboard <-> RGW credentials 2024-03-26T02:10:04.194 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:03 smithi052 ceph-mon[25889]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 294 KiB/s wr, 18 op/s 2024-03-26T02:10:04.194 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:03 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2188108003' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:10:04.194 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:03 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/246062740' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:10:04.194 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:03 smithi052 ceph-mon[25889]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T02:10:04.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:03 smithi165 ceph-mon[27627]: Checking dashboard <-> RGW credentials 2024-03-26T02:10:04.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:03 smithi165 ceph-mon[27627]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 294 KiB/s wr, 18 op/s 2024-03-26T02:10:04.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:03 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2188108003' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:10:04.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:03 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/246062740' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:10:04.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:03 smithi165 ceph-mon[27627]: osdmap e23: 3 total, 3 up, 3 in 2024-03-26T02:10:04.439 DEBUG:teuthology.orchestra.run.smithi052:node-exporter.a> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T02:10:04.441 DEBUG:teuthology.orchestra.run.smithi165:node-exporter.b> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T02:10:04.444 DEBUG:teuthology.orchestra.run.smithi191:node-exporter.c> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.c.service 2024-03-26T02:10:04.446 INFO:tasks.cephadm:Adding alertmanager.a on smithi052 2024-03-26T02:10:04.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi052=a' 2024-03-26T02:10:04.713 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:10:05.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:04 smithi052 ceph-mon[25889]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi052=a;smithi165=b;smithi191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:05.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:04 smithi052 ceph-mon[25889]: Saving service node-exporter spec with placement smithi052=a;smithi165=b;smithi191=c;count:3 2024-03-26T02:10:05.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:04 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:05.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:04 smithi052 ceph-mon[25889]: pgmap v110: 33 pgs: 11 active+clean, 22 unknown; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 6.1 KiB/s rd, 294 KiB/s wr, 19 op/s 2024-03-26T02:10:05.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:04 smithi052 ceph-mon[25889]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T02:10:05.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:04 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:10:05.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:04 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:10:05.141 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:04 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:05.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:04 smithi191 ceph-mon[27701]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi052=a;smithi165=b;smithi191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:05.149 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:04 smithi191 ceph-mon[27701]: Saving service node-exporter spec with placement smithi052=a;smithi165=b;smithi191=c;count:3 2024-03-26T02:10:05.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:04 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:05.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:04 smithi191 ceph-mon[27701]: pgmap v110: 33 pgs: 11 active+clean, 22 unknown; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 6.1 KiB/s rd, 294 KiB/s wr, 19 op/s 2024-03-26T02:10:05.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:04 smithi191 ceph-mon[27701]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T02:10:05.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:04 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:10:05.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:04 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:10:05.150 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:04 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:05.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:04 smithi165 ceph-mon[27627]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi052=a;smithi165=b;smithi191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:05.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:04 smithi165 ceph-mon[27627]: Saving service node-exporter spec with placement smithi052=a;smithi165=b;smithi191=c;count:3 2024-03-26T02:10:05.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:04 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:05.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:04 smithi165 ceph-mon[27627]: pgmap v110: 33 pgs: 11 active+clean, 22 unknown; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 6.1 KiB/s rd, 294 KiB/s wr, 19 op/s 2024-03-26T02:10:05.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:04 smithi165 ceph-mon[27627]: osdmap e24: 3 total, 3 up, 3 in 2024-03-26T02:10:05.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:04 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:10:05.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:04 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:10:05.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:04 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:06 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:10:06.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:06 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:10:06.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:06 smithi165 ceph-mon[27627]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T02:10:06.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:06 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:06 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:06 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:06 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:06 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:10:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:06 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:10:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:06 smithi052 ceph-mon[25889]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T02:10:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:06 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:06 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:06 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:06 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:06 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:10:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:06 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:10:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:06 smithi191 ceph-mon[27701]: osdmap e25: 3 total, 3 up, 3 in 2024-03-26T02:10:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:06 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:06 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:06 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:06 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:06.557 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled alertmanager update... 2024-03-26T02:10:07.192 DEBUG:teuthology.orchestra.run.smithi052:alertmanager.a> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T02:10:07.195 INFO:tasks.cephadm:Adding grafana.a on smithi165 2024-03-26T02:10:07.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi165=a' 2024-03-26T02:10:07.430 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:10:07.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:07 smithi165 ceph-mon[27627]: pgmap v113: 65 pgs: 9 creating+peering, 31 unknown, 25 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T02:10:07.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:07 smithi165 ceph-mon[27627]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T02:10:07.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:07 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:10:07.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:07 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:10:07.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:07 smithi165 ceph-mon[27627]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi052=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:07.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:07 smithi165 ceph-mon[27627]: Saving service alertmanager spec with placement smithi052=a;count:1 2024-03-26T02:10:07.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:07 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:07.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:07 smithi052 ceph-mon[25889]: pgmap v113: 65 pgs: 9 creating+peering, 31 unknown, 25 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T02:10:07.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:07 smithi052 ceph-mon[25889]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T02:10:07.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:07 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:10:07.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:07 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:10:07.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:07 smithi052 ceph-mon[25889]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi052=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:07.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:07 smithi052 ceph-mon[25889]: Saving service alertmanager spec with placement smithi052=a;count:1 2024-03-26T02:10:07.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:07 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:07 smithi191 ceph-mon[27701]: pgmap v113: 65 pgs: 9 creating+peering, 31 unknown, 25 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T02:10:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:07 smithi191 ceph-mon[27701]: osdmap e26: 3 total, 3 up, 3 in 2024-03-26T02:10:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:07 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:10:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:07 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:10:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:07 smithi191 ceph-mon[27701]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi052=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:07 smithi191 ceph-mon[27701]: Saving service alertmanager spec with placement smithi052=a;count:1 2024-03-26T02:10:07.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:07 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:08.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:08 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:10:08.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:08 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:10:08.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:08 smithi165 ceph-mon[27627]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T02:10:08.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:08 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:10:08.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:08 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:10:08.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:08 smithi052 ceph-mon[25889]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T02:10:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:08 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:10:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:08 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:10:08.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:08 smithi191 ceph-mon[27701]: osdmap e27: 3 total, 3 up, 3 in 2024-03-26T02:10:09.284 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled grafana update... 2024-03-26T02:10:09.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:09 smithi165 ceph-mon[27627]: pgmap v116: 97 pgs: 20 creating+peering, 32 unknown, 45 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T02:10:09.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:09 smithi165 ceph-mon[27627]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T02:10:09.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:09 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:10:09.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:09 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:10:09.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:09 smithi052 ceph-mon[25889]: pgmap v116: 97 pgs: 20 creating+peering, 32 unknown, 45 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T02:10:09.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:09 smithi052 ceph-mon[25889]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T02:10:09.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:09 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:10:09.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:09 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:10:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:09 smithi191 ceph-mon[27701]: pgmap v116: 97 pgs: 20 creating+peering, 32 unknown, 45 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T02:10:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:09 smithi191 ceph-mon[27701]: osdmap e28: 3 total, 3 up, 3 in 2024-03-26T02:10:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:09 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:10:09.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:09 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:10:09.842 DEBUG:teuthology.orchestra.run.smithi165:grafana.a> sudo journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T02:10:09.845 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T02:10:09.846 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T02:10:10.097 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:10.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:10 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:10:10.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:10 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:10:10.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:10 smithi052 ceph-mon[25889]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T02:10:10.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:10 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:10:10.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:10 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:10:10.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:10 smithi052 ceph-mon[25889]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:10.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:10 smithi052 ceph-mon[25889]: Saving service grafana spec with placement smithi165=a;count:1 2024-03-26T02:10:10.440 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:10 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:10.441 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:10 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:10:10.441 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:10 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:10:10.441 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:10 smithi052 ceph-mon[25889]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T02:10:10.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:10 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:10:10.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:10 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:10:10.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:10 smithi165 ceph-mon[27627]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T02:10:10.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:10 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:10:10.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:10 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:10:10.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:10 smithi165 ceph-mon[27627]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:10.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:10 smithi165 ceph-mon[27627]: Saving service grafana spec with placement smithi165=a;count:1 2024-03-26T02:10:10.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:10 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:10.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:10 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:10:10.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:10 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:10:10.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:10 smithi165 ceph-mon[27627]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T02:10:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:10 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:10:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:10 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:10:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:10 smithi191 ceph-mon[27701]: osdmap e29: 3 total, 3 up, 3 in 2024-03-26T02:10:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:10 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:10:10.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:10 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:10:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:10 smithi191 ceph-mon[27701]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:10 smithi191 ceph-mon[27701]: Saving service grafana spec with placement smithi165=a;count:1 2024-03-26T02:10:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:10 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:10 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3568332277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:10:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:10 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2640559852' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:10:10.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:10 smithi191 ceph-mon[27701]: osdmap e30: 3 total, 3 up, 3 in 2024-03-26T02:10:11.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:11 smithi165 ceph-mon[27627]: pgmap v119: 129 pgs: 33 creating+peering, 31 unknown, 65 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:10:11.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:11 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:11.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:11 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:11.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:11 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:11.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:11 smithi165 ceph-mon[27627]: Deploying daemon prometheus.a on smithi165 2024-03-26T02:10:11.595 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:11 smithi052 ceph-mon[25889]: pgmap v119: 129 pgs: 33 creating+peering, 31 unknown, 65 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:10:11.595 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:11 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:11.596 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:11 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:11.596 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:11 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:11.596 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:11 smithi052 ceph-mon[25889]: Deploying daemon prometheus.a on smithi165 2024-03-26T02:10:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:11 smithi191 ceph-mon[27701]: pgmap v119: 129 pgs: 33 creating+peering, 31 unknown, 65 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:10:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:11 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:11 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:11 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:11.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:11 smithi191 ceph-mon[27701]: Deploying daemon prometheus.a on smithi165 2024-03-26T02:10:12.149 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2024-03-26T02:10:12.150 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQCELgJmn9LQCBAAWIZaVnqdQQEUBZo4bYnEZw== 2024-03-26T02:10:12.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:12 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/1764592078' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:12.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:12 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/1764592078' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:10:12.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:12 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/1764592078' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:12.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:12 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/1764592078' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:10:12.773 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:10:12.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T02:10:12.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T02:10:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:12 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/1764592078' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:12.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:12 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/1764592078' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:10:12.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T02:10:13.420 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:10:13.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:13 smithi165 ceph-mon[27627]: pgmap v121: 129 pgs: 13 creating+peering, 116 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 57 KiB/s rd, 2.7 KiB/s wr, 125 op/s 2024-03-26T02:10:13.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:13 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:13.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:13 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:13.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:13 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:13.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:13 smithi052 ceph-mon[25889]: pgmap v121: 129 pgs: 13 creating+peering, 116 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 57 KiB/s rd, 2.7 KiB/s wr, 125 op/s 2024-03-26T02:10:13.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:13 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:13.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:13 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:13.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:13 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:13 smithi191 ceph-mon[27701]: pgmap v121: 129 pgs: 13 creating+peering, 116 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 57 KiB/s rd, 2.7 KiB/s wr, 125 op/s 2024-03-26T02:10:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:13 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:13 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:13.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:13 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:15.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:15 smithi165 ceph-mon[27627]: pgmap v122: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 4.0 KiB/s wr, 182 op/s 2024-03-26T02:10:15.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:15 smithi052 ceph-mon[25889]: pgmap v122: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 4.0 KiB/s wr, 182 op/s 2024-03-26T02:10:15.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:15 smithi191 ceph-mon[27701]: pgmap v122: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 4.0 KiB/s wr, 182 op/s 2024-03-26T02:10:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:16 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:16 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:16.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:16 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:16.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:16 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:16.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:16 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:16.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:16 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:16.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:16 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:16.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:16 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:16.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:16 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:17 smithi191 ceph-mon[27701]: pgmap v123: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 92 KiB/s rd, 4.6 KiB/s wr, 205 op/s 2024-03-26T02:10:17.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:17 smithi165 ceph-mon[27627]: pgmap v123: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 92 KiB/s rd, 4.6 KiB/s wr, 205 op/s 2024-03-26T02:10:17.862 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:17 smithi052 ceph-mon[25889]: pgmap v123: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 92 KiB/s rd, 4.6 KiB/s wr, 205 op/s 2024-03-26T02:10:18.958 INFO:teuthology.orchestra.run.smithi165.stdout:[client.1] 2024-03-26T02:10:18.958 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCKLgJmMq7YOBAAGButbNqYoIhGIJ8tLmTF1w== 2024-03-26T02:10:19.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:19 smithi165 ceph-mon[27627]: pgmap v124: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 4.0 KiB/s wr, 181 op/s 2024-03-26T02:10:19.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:19 smithi165 ceph-mon[27627]: from='client.? 172.21.15.165:0/3190669596' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:19.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:19 smithi165 ceph-mon[27627]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:19.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:19 smithi165 ceph-mon[27627]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:10:19.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:19 smithi052 ceph-mon[25889]: pgmap v124: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 4.0 KiB/s wr, 181 op/s 2024-03-26T02:10:19.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:19 smithi052 ceph-mon[25889]: from='client.? 172.21.15.165:0/3190669596' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:19.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:19 smithi052 ceph-mon[25889]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:19.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:19 smithi052 ceph-mon[25889]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:10:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:19 smithi191 ceph-mon[27701]: pgmap v124: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 4.0 KiB/s wr, 181 op/s 2024-03-26T02:10:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:19 smithi191 ceph-mon[27701]: from='client.? 172.21.15.165:0/3190669596' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:19 smithi191 ceph-mon[27701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:19.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:19 smithi191 ceph-mon[27701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:10:20.071 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-26T02:10:20.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T02:10:20.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T02:10:20.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T02:10:20.365 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T02:10:21.758 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:21 smithi191 ceph-mon[27701]: pgmap v125: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 72 KiB/s rd, 3.5 KiB/s wr, 159 op/s 2024-03-26T02:10:21.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:21 smithi165 ceph-mon[27627]: pgmap v125: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 72 KiB/s rd, 3.5 KiB/s wr, 159 op/s 2024-03-26T02:10:21.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:21 smithi052 ceph-mon[25889]: pgmap v125: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 72 KiB/s rd, 3.5 KiB/s wr, 159 op/s 2024-03-26T02:10:22.253 INFO:teuthology.orchestra.run.smithi191.stdout:[client.2] 2024-03-26T02:10:22.254 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCOLgJmzmfXDhAA2nvdjosRFwnPUxruCSYLPg== 2024-03-26T02:10:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:22 smithi191 ceph-mon[27701]: from='client.? 172.21.15.191:0/1956828261' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:22 smithi191 ceph-mon[27701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:22.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:22 smithi191 ceph-mon[27701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:10:22.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-mon[27627]: from='client.? 172.21.15.191:0/1956828261' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:22.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-mon[27627]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:22.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-mon[27627]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:10:22.805 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 podman[35977]: 2024-03-26 02:10:22.689094672 +0000 UTC m=+0.132880972 container create 51f40423f9a900dcb805a09274ab29662a6545c45559ed2fb4e8ff1d2de51af0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:10:22.935 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-26T02:10:22.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-26T02:10:22.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-26T02:10:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:22 smithi052 ceph-mon[25889]: from='client.? 172.21.15.191:0/1956828261' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:22 smithi052 ceph-mon[25889]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:10:22.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:22 smithi052 ceph-mon[25889]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:10:22.980 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T02:10:22.981 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T02:10:22.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T02:10:23.203 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 podman[35977]: 2024-03-26 02:10:22.899720418 +0000 UTC m=+0.343506719 container init 51f40423f9a900dcb805a09274ab29662a6545c45559ed2fb4e8ff1d2de51af0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:10:23.203 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 podman[35977]: 2024-03-26 02:10:22.904840998 +0000 UTC m=+0.348627299 container start 51f40423f9a900dcb805a09274ab29662a6545c45559ed2fb4e8ff1d2de51af0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:10:23.203 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.928Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T02:10:23.203 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.928Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T02:10:23.203 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.928Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi165 (none))" 2024-03-26T02:10:23.203 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.928Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:10:23.203 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.928Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:10:23.203 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.929Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.929Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.931Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.931Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.933Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.933Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.378µs 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.933Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.933Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.933Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=29.925µs wal_replay_duration=315.848µs wbl_replay_duration=155ns total_replay_duration=425.851µs 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.934Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.934Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.934Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.952Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.835665ms db_storage=750ns remote_storage=1.503µs web_handler=409ns query_engine=528ns scrape=425.803µs scrape_sd=71.001µs notify=706ns notify_sd=1.309µs rules=17.095796ms tracing=5.46µs 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.952Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:10:22.952Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T02:10:23.204 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 bash[35977]: 51f40423f9a900dcb805a09274ab29662a6545c45559ed2fb4e8ff1d2de51af0 2024-03-26T02:10:23.205 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:10:22 smithi165 systemd[1]: Started Ceph prometheus.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:10:23.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:23 smithi191 ceph-mon[27701]: pgmap v126: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 61 KiB/s rd, 3.0 KiB/s wr, 135 op/s 2024-03-26T02:10:23.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:23 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:23.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:23 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:23.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:23 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:23.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:23 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:10:23.873 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:23.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:23 smithi165 ceph-mon[27627]: pgmap v126: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 61 KiB/s rd, 3.0 KiB/s wr, 135 op/s 2024-03-26T02:10:23.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:23 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:23.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:23 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:23.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:23 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:23.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:23 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:10:23.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:23 smithi052 ceph-mon[25889]: pgmap v126: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 61 KiB/s rd, 3.0 KiB/s wr, 135 op/s 2024-03-26T02:10:23.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:23 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:23.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:23 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:23.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:23 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' 2024-03-26T02:10:23.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:23 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:10:24.317 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:24 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ignoring --setuser ceph since I am not root 2024-03-26T02:10:24.317 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:24 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ignoring --setgroup ceph since I am not root 2024-03-26T02:10:24.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:24 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: ignoring --setuser ceph since I am not root 2024-03-26T02:10:24.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:24 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: ignoring --setgroup ceph since I am not root 2024-03-26T02:10:24.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:24 smithi165 ceph-mgr[29353]: -- 172.21.15.165:0/1419386149 <== mon.2 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5558483661a0 con 0x55584833cc00 2024-03-26T02:10:24.537 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:24 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: ignoring --setuser ceph since I am not root 2024-03-26T02:10:24.537 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:24 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: ignoring --setgroup ceph since I am not root 2024-03-26T02:10:24.537 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:24 smithi191 ceph-mgr[30409]: -- 172.21.15.191:0/2721809198 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e1c37d41a0 con 0x55e1c37ab000 2024-03-26T02:10:24.952 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:24 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:24.649+0000 7f90f3c3b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:10:24.952 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:24 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:24.742+0000 7f90f3c3b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:10:24.960 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:24 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:24.687+0000 7ff494510200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:10:24.960 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:24 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:24.787+0000 7ff494510200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:10:25.037 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:24 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:24.652+0000 7faea45fd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:10:25.037 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:24 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:24.745+0000 7faea45fd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:10:25.375 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:25 smithi191 ceph-mon[27701]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:10:25.376 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:25 smithi191 ceph-mon[27701]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T02:10:25.388 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:25 smithi165 ceph-mon[27627]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:10:25.388 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:25 smithi165 ceph-mon[27627]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T02:10:25.411 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:25 smithi052 ceph-mon[25889]: from='mgr.14152 172.21.15.52:0/446086192' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:10:25.411 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:25 smithi052 ceph-mon[25889]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-26T02:10:25.702 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:25 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:25.387+0000 7f90f3c3b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:10:25.702 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:25 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:25.481+0000 7f90f3c3b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:10:25.702 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:25 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:25.627+0000 7f90f3c3b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:10:25.710 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:25 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:25.472+0000 7ff494510200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:10:25.710 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:25 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:25.568+0000 7ff494510200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:10:25.713 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:25 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:25.374+0000 7faea45fd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:10:25.713 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:25 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:25.467+0000 7faea45fd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:10:25.713 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:25 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:25.612+0000 7faea45fd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:10:25.713 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:25 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:25.712+0000 7faea45fd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:10:25.966 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:25 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:25.800+0000 7faea45fd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:10:26.025 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:25 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:25.712+0000 7ff494510200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:10:26.025 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:25 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:25.832+0000 7ff494510200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:10:26.025 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:25 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:25.920+0000 7ff494510200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:10:26.076 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:25 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:25.723+0000 7f90f3c3b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:10:26.076 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:25 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:25.809+0000 7f90f3c3b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:10:26.076 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:26.074+0000 7f90f3c3b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:10:26.281 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:26 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:26.078+0000 7faea45fd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:10:26.281 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:26 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:26.192+0000 7faea45fd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:10:26.282 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:26 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:26.280+0000 7faea45fd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:10:26.291 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:26 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:26.202+0000 7ff494510200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:10:26.366 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:26.183+0000 7f90f3c3b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:10:26.366 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:26.269+0000 7f90f3c3b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:10:26.366 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:26.364+0000 7f90f3c3b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:10:26.537 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:26 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:26.375+0000 7faea45fd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:10:26.686 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:10:26.687 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:26 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:26.321+0000 7ff494510200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:10:26.687 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:26 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:26.416+0000 7ff494510200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:10:26.687 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:26 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:26.526+0000 7ff494510200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:10:26.702 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:26.541+0000 7f90f3c3b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:10:26.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:26 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2738042221' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:10:26.961 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:26 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:26.711+0000 7ff494510200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:10:27.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:26 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2738042221' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:10:27.037 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:26 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:26.554+0000 7faea45fd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:10:27.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:26 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2738042221' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:10:27.725 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":1692469977},{"type":"v1","addr":"172.21.15.52:6801","nonce":1692469977}]},"active_addr":"172.21.15.52:6801/1692469977","active_change":"2024-03-26T02:07:14.675976+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.52:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":4187790001}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2111434236}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":33168520}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2705564098}]}]} 2024-03-26T02:10:27.732 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T02:10:27.732 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T02:10:27.732 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:10:27.787 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:27 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:27.509+0000 7faea45fd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:10:27.787 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:27 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:27.638+0000 7faea45fd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:10:27.952 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:27 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:27.510+0000 7f90f3c3b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:10:27.952 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:27 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:27.643+0000 7f90f3c3b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:10:27.960 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:27 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:27.714+0000 7ff494510200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:10:27.960 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:27 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:27.854+0000 7ff494510200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:10:27.976 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:28.257 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:27 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:27.972+0000 7faea45fd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:10:28.257 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:28.061+0000 7faea45fd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:10:28.257 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:28.256+0000 7faea45fd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:10:28.284 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:27 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:27.994+0000 7f90f3c3b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:10:28.284 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:28 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:28.086+0000 7f90f3c3b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:10:28.418 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:28 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:28.213+0000 7ff494510200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:10:28.418 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:28 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:28.306+0000 7ff494510200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:10:28.537 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:28.440+0000 7faea45fd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:10:28.702 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:28 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:28.282+0000 7f90f3c3b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:10:28.702 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:28 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:28.470+0000 7f90f3c3b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:10:28.709 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:28 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:28.513+0000 7ff494510200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:10:28.960 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:28 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:28.709+0000 7ff494510200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:10:28.983 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:28.722+0000 7faea45fd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:10:28.983 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:28.982+0000 7faea45fd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:10:29.038 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:28 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:28.759+0000 7f90f3c3b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:10:29.273 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:29.005+0000 7ff494510200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:10:29.286 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:29 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:29.217+0000 7faea45fd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:10:29.291 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:29 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:29.037+0000 7f90f3c3b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:10:29.525 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:29.272+0000 7ff494510200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:10:29.526 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:29.524+0000 7ff494510200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:10:29.702 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:29 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:29.289+0000 7f90f3c3b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:10:29.797 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:10:29.797 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":30,"fsid":"46f2b2dc-eb15-11ee-95d0-87774f69a715","created":"2024-03-26T02:06:16.299492+0000","modified":"2024-03-26T02:10:10.161344+0000","last_up_change":"2024-03-26T02:09:49.511019+0000","last_in_change":"2024-03-26T02:09:28.702724+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T02:09:51.026808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T02:10:01.511153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-26T02:10:03.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-26T02:10:05.417212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-26T02:10:07.472076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67c1a134-0690-420c-b67f-fb0239b4361f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6803","nonce":781166251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6805","nonce":781166251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6809","nonce":781166251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6807","nonce":781166251}]},"public_addr":"172.21.15.52:6803/781166251","cluster_addr":"172.21.15.52:6805/781166251","heartbeat_back_addr":"172.21.15.52:6809/781166251","heartbeat_front_addr":"172.21.15.52:6807/781166251","state":["exists","up"]},{"osd":1,"uuid":"b9ecdb53-2378-496c-adf5-2fb53c8ddc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6801","nonce":3456552919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6803","nonce":3456552919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6807","nonce":3456552919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6805","nonce":3456552919}]},"public_addr":"172.21.15.165:6801/3456552919","cluster_addr":"172.21.15.165:6803/3456552919","heartbeat_back_addr":"172.21.15.165:6807/3456552919","heartbeat_front_addr":"172.21.15.165:6805/3456552919","state":["exists","up"]},{"osd":2,"uuid":"896c1548-6eb5-4713-a7f9-39f4e7e6c7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6801","nonce":945058458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6803","nonce":945058458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6807","nonce":945058458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6805","nonce":945058458}]},"public_addr":"172.21.15.191:6801/945058458","cluster_addr":"172.21.15.191:6803/945058458","heartbeat_back_addr":"172.21.15.191:6807/945058458","heartbeat_front_addr":"172.21.15.191:6805/945058458","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:08:53.708991+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:09:20.587130+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:09:47.048074+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/3063795633":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/808014555":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/4225488723":"2024-03-27T02:06:47.789796+0000","172.21.15.52:6801/1514360646":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/201621324":"2024-03-27T02:06:47.789796+0000","172.21.15.52:0/3458856682":"2024-03-27T02:06:47.789796+0000","172.21.15.52:6800/1514360646":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/426412327":"2024-03-27T02:07:14.675819+0000","172.21.15.52:6800/1384523876":"2024-03-27T02:06:47.789796+0000","172.21.15.52:6801/1384523876":"2024-03-27T02:06:47.789796+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:10:30.111 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:29 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/1346049203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:10:30.122 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:29 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/1346049203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:10:30.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:29 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/1346049203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:10:30.287 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:30 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:30.012+0000 7faea45fd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:10:30.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:30 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:30.121+0000 7f90f3c3b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:10:30.457 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T02:10:30.458 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:10:30.703 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:30.710 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:30 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:30.383+0000 7ff494510200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:10:30.892 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:30 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:30.579+0000 7faea45fd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:10:30.892 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:30 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:30.676+0000 7faea45fd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:10:30.892 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:30 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:30.891+0000 7faea45fd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:10:30.952 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:30 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:30.699+0000 7f90f3c3b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:10:30.952 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:30 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:30.798+0000 7f90f3c3b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:10:31.081 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:30 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:30.992+0000 7ff494510200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:10:31.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:30 smithi052 ceph-mon[25889]: Standby manager daemon z restarted 2024-03-26T02:10:31.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:30 smithi052 ceph-mon[25889]: Standby manager daemon z started 2024-03-26T02:10:31.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:30 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T02:10:31.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:30 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:10:31.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:30 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T02:10:31.082 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:30 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:10:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:30 smithi191 ceph-mon[27701]: Standby manager daemon z restarted 2024-03-26T02:10:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:30 smithi191 ceph-mon[27701]: Standby manager daemon z started 2024-03-26T02:10:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:30 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T02:10:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:30 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:10:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:30 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T02:10:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:30 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:10:31.288 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:30 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: [26/Mar/2024:02:10:30] ENGINE Bus STARTING 2024-03-26T02:10:31.288 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: CherryPy Checker: 2024-03-26T02:10:31.288 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: The Application mounted at '' has an empty config. 2024-03-26T02:10:31.288 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: 2024-03-26T02:10:31.288 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: [26/Mar/2024:02:10:31] ENGINE Serving on http://:::9283 2024-03-26T02:10:31.288 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z[30346]: [26/Mar/2024:02:10:31] ENGINE Bus STARTED 2024-03-26T02:10:31.438 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:31.095+0000 7ff494510200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:10:31.439 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:31.323+0000 7ff494510200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:10:31.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:30 smithi165 ceph-mon[27627]: Standby manager daemon z restarted 2024-03-26T02:10:31.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:30 smithi165 ceph-mon[27627]: Standby manager daemon z started 2024-03-26T02:10:31.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:30 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T02:10:31.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:30 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:10:31.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:30 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T02:10:31.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:30 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.191:0/1833563480' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:10:31.453 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:31.017+0000 7f90f3c3b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:10:31.453 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: [26/Mar/2024:02:10:31] ENGINE Bus STARTING 2024-03-26T02:10:31.453 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: CherryPy Checker: 2024-03-26T02:10:31.453 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: The Application mounted at '' has an empty config. 2024-03-26T02:10:31.453 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: 2024-03-26T02:10:31.453 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: [26/Mar/2024:02:10:31] ENGINE Serving on http://:::9283 2024-03-26T02:10:31.453 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x[29325]: [26/Mar/2024:02:10:31] ENGINE Bus STARTED 2024-03-26T02:10:31.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:10:31] ENGINE Bus STARTING 2024-03-26T02:10:31.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: CherryPy Checker: 2024-03-26T02:10:31.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: The Application mounted at '' has an empty config. 2024-03-26T02:10:31.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:10:31.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:10:31] ENGINE Serving on http://:::9283 2024-03-26T02:10:31.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:10:31] ENGINE Bus STARTED 2024-03-26T02:10:32.198 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-26T02:10:32.198 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: Standby manager daemon x restarted 2024-03-26T02:10:32.198 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: Standby manager daemon x started 2024-03-26T02:10:32.198 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:10:32.198 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:10:32.198 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:10:32.198 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:10:32.198 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: Active manager daemon y restarted 2024-03-26T02:10:32.198 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: Activating manager daemon y 2024-03-26T02:10:32.198 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: mgrmap e19: y(active, starting, since 0.00968486s), standbys: x, z 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:10:32.199 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: Manager daemon y is now available 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]': finished 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]': finished 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]': finished 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:32.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:32.201 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:32.201 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:10:32.201 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:31 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:10:32.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-26T02:10:32.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: Standby manager daemon x restarted 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: Standby manager daemon x started 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: Active manager daemon y restarted 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: Activating manager daemon y 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: mgrmap e19: y(active, starting, since 0.00968486s), standbys: x, z 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:10:32.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: Manager daemon y is now available 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:10:32.289 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:10:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]': finished 2024-03-26T02:10:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]': finished 2024-03-26T02:10:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]': finished 2024-03-26T02:10:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:10:32.290 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:31 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:10:32.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: Standby manager daemon x restarted 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: Standby manager daemon x started 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.? 172.21.15.165:0/1934196669' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: Active manager daemon y restarted 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: Activating manager daemon y 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: osdmap e31: 3 total, 3 up, 3 in 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: mgrmap e19: y(active, starting, since 0.00968486s), standbys: x, z 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:10:32.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: Manager daemon y is now available 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:10:32.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:10:32.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]': finished 2024-03-26T02:10:32.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]': finished 2024-03-26T02:10:32.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]': finished 2024-03-26T02:10:32.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:32.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:32.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:32.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:32.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:10:32.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:31 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:10:32.703 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:10:32.703 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":31,"fsid":"46f2b2dc-eb15-11ee-95d0-87774f69a715","created":"2024-03-26T02:06:16.299492+0000","modified":"2024-03-26T02:10:31.337195+0000","last_up_change":"2024-03-26T02:09:49.511019+0000","last_in_change":"2024-03-26T02:09:28.702724+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T02:09:51.026808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T02:10:01.511153+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-26T02:10:03.455328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-26T02:10:05.417212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-26T02:10:07.472076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67c1a134-0690-420c-b67f-fb0239b4361f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6803","nonce":781166251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6805","nonce":781166251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6809","nonce":781166251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":781166251},{"type":"v1","addr":"172.21.15.52:6807","nonce":781166251}]},"public_addr":"172.21.15.52:6803/781166251","cluster_addr":"172.21.15.52:6805/781166251","heartbeat_back_addr":"172.21.15.52:6809/781166251","heartbeat_front_addr":"172.21.15.52:6807/781166251","state":["exists","up"]},{"osd":1,"uuid":"b9ecdb53-2378-496c-adf5-2fb53c8ddc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6801","nonce":3456552919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6803","nonce":3456552919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6807","nonce":3456552919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3456552919},{"type":"v1","addr":"172.21.15.165:6805","nonce":3456552919}]},"public_addr":"172.21.15.165:6801/3456552919","cluster_addr":"172.21.15.165:6803/3456552919","heartbeat_back_addr":"172.21.15.165:6807/3456552919","heartbeat_front_addr":"172.21.15.165:6805/3456552919","state":["exists","up"]},{"osd":2,"uuid":"896c1548-6eb5-4713-a7f9-39f4e7e6c7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6801","nonce":945058458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6803","nonce":945058458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6807","nonce":945058458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":945058458},{"type":"v1","addr":"172.21.15.191:6805","nonce":945058458}]},"public_addr":"172.21.15.191:6801/945058458","cluster_addr":"172.21.15.191:6803/945058458","heartbeat_back_addr":"172.21.15.191:6807/945058458","heartbeat_front_addr":"172.21.15.191:6805/945058458","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:08:53.708991+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:09:20.587130+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:09:47.048074+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/2111434236":"2024-03-27T02:10:31.337150+0000","172.21.15.52:6800/1692469977":"2024-03-27T02:10:31.337150+0000","172.21.15.52:6801/1692469977":"2024-03-27T02:10:31.337150+0000","172.21.15.52:0/2705564098":"2024-03-27T02:10:31.337150+0000","172.21.15.52:0/4187790001":"2024-03-27T02:10:31.337150+0000","172.21.15.52:0/33168520":"2024-03-27T02:10:31.337150+0000","172.21.15.52:0/3063795633":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/808014555":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/4225488723":"2024-03-27T02:06:47.789796+0000","172.21.15.52:6801/1514360646":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/201621324":"2024-03-27T02:06:47.789796+0000","172.21.15.52:0/3458856682":"2024-03-27T02:06:47.789796+0000","172.21.15.52:6800/1514360646":"2024-03-27T02:07:14.675819+0000","172.21.15.52:0/426412327":"2024-03-27T02:07:14.675819+0000","172.21.15.52:6800/1384523876":"2024-03-27T02:06:47.789796+0000","172.21.15.52:6801/1384523876":"2024-03-27T02:06:47.789796+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:10:33.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-26T02:10:33.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-26T02:10:33.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: Updating smithi052:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: mgrmap e20: y(active, since 1.01614s), standbys: x, z 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/2278332849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.328 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-26T02:10:33.328 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-26T02:10:33.328 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: Updating smithi052:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: mgrmap e20: y(active, since 1.01614s), standbys: x, z 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/2278332849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.329 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T02:10:33.356 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T02:10:33.357 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T02:10:33.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-26T02:10:33.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-26T02:10:33.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-26T02:10:33.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: Updating smithi052:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:10:33.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:10:33.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:10:33.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: mgrmap e20: y(active, since 1.01614s), standbys: x, z 2024-03-26T02:10:33.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/2278332849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:10:33.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:33.628 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:33.632 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:33.638 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:34.023 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: [26/Mar/2024:02:10:32] ENGINE Bus STARTING 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: [26/Mar/2024:02:10:32] ENGINE Serving on http://172.21.15.52:8765 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: [26/Mar/2024:02:10:32] ENGINE Serving on https://172.21.15.52:7151 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: [26/Mar/2024:02:10:32] ENGINE Bus STARTED 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: Updating smithi052:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]: dispatch 2024-03-26T02:10:34.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:33 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-26T02:10:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: [26/Mar/2024:02:10:32] ENGINE Bus STARTING 2024-03-26T02:10:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:10:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:10:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:10:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: [26/Mar/2024:02:10:32] ENGINE Serving on http://172.21.15.52:8765 2024-03-26T02:10:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: [26/Mar/2024:02:10:32] ENGINE Serving on https://172.21.15.52:7151 2024-03-26T02:10:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: [26/Mar/2024:02:10:32] ENGINE Bus STARTED 2024-03-26T02:10:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: Updating smithi052:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:10:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:10:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:10:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:10:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]: dispatch 2024-03-26T02:10:34.288 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:33 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-26T02:10:34.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: [26/Mar/2024:02:10:32] ENGINE Bus STARTING 2024-03-26T02:10:34.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: [26/Mar/2024:02:10:32] ENGINE Serving on http://172.21.15.52:8765 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: [26/Mar/2024:02:10:32] ENGINE Serving on https://172.21.15.52:7151 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: [26/Mar/2024:02:10:32] ENGINE Bus STARTED 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: Updating smithi052:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: Updating smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: Updating smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]: dispatch 2024-03-26T02:10:34.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:33 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-26T02:10:35.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:34 smithi052 ceph-mon[25889]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:35.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:34 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:35.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:34 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:10:35.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:34 smithi052 ceph-mon[25889]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-26T02:10:35.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:34 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:35.227 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:34 smithi165 ceph-mon[27627]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:35.227 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:34 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:35.227 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:34 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:10:35.227 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:34 smithi165 ceph-mon[27627]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-26T02:10:35.227 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:34 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:34 smithi191 ceph-mon[27701]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:34 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:34 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:10:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:34 smithi191 ceph-mon[27701]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-26T02:10:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:34 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:36.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:35 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:36.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:35 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:36.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:35 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:36.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:35 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:36.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:35 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:10:36.087 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:35 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:36.087 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:35 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:36.087 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:35 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:36.087 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:35 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:36.088 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:35 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:10:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:35 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:35 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:35 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:35 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:36.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:35 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:10:37.037 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:36 smithi052 ceph-mon[25889]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:37.038 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:36 smithi052 ceph-mon[25889]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-26T02:10:37.038 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:36 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:37.038 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:36 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:37.038 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:36 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:37.038 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:36 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:10:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:36 smithi191 ceph-mon[27701]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:36 smithi191 ceph-mon[27701]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-26T02:10:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:36 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:36 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:36 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:37.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:36 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:10:37.441 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:36 smithi165 ceph-mon[27627]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:37.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:36 smithi165 ceph-mon[27627]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-26T02:10:37.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:36 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:37.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:36 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:37.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:36 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:37.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:36 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi191", "caps": []}]: dispatch 2024-03-26T02:10:37.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:37 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ::ffff:172.21.15.165 - - [26/Mar/2024:02:10:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:10:38.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:37 smithi165 ceph-mon[27627]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:10:38.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:37 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:38.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:37 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:38.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:37 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:38.196 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:37 smithi191 ceph-mon[27701]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:10:38.196 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:37 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:38.196 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:37 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:38.196 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:37 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:38.242 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:37 smithi052 ceph-mon[25889]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:10:38.243 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:37 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:38.243 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:37 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:38.243 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:37 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.167 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:38 smithi052 ceph-mon[25889]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:39.167 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:38 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.168 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:38 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.168 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:38 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.168 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:38 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.168 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:38 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.168 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:38 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.195 INFO:teuthology.orchestra.run.smithi052.stdout:34359738391 2024-03-26T02:10:39.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T02:10:39.204 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:38 smithi191 ceph-mon[27701]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:39.204 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:38 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.204 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:38 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.205 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:38 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.205 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:38 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.205 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:38 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.205 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:38 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:38 smithi165 ceph-mon[27627]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:39.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:38 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:38 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:38 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:38 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:38 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:38 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:39.747 INFO:teuthology.orchestra.run.smithi052.stdout:77309411340 2024-03-26T02:10:39.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T02:10:39.889 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:40.276 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:40 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.276 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:40 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.276 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:40 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.276 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:40 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.276 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:40 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.276 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:40 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.352 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:40 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.352 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:40 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.352 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:40 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.352 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:40 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.352 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:40 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.352 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:40 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:40 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:40 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:40 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:40 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:40 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:40.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:40 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:41 smithi165 ceph-mon[27627]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:10:41.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:41 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:41 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:41 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:41 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:41 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:41 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.358 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:41 smithi052 ceph-mon[25889]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:10:41.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:41 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:41 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:41 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:41 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:41 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.359 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:41 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:41 smithi191 ceph-mon[27701]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:10:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:41 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:41 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:41 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:41 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:41 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:41.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:41 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:42.792 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:42 smithi052 ceph-mon[25889]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:10:42.792 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:42 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:42.792 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:42 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:42.793 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:42 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:42.825 INFO:teuthology.orchestra.run.smithi052.stdout:55834574866 2024-03-26T02:10:42.825 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T02:10:42.952 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:42 smithi191 ceph-mon[27701]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:10:42.952 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:42 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:42.952 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:42 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:42.952 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:42 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:43.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:42 smithi165 ceph-mon[27627]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:10:43.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:42 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:43.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:42 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:43.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:42 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:43.559 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:43.777 INFO:teuthology.orchestra.run.smithi052.stdout:34359738391 2024-03-26T02:10:44.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:10:44.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:44.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:44.636 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:44.637 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.637 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: Deploying daemon alertmanager.a on smithi052 2024-03-26T02:10:44.637 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:44 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/3407891642' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T02:10:44.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:10:44.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:44.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:44.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:44.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: Deploying daemon alertmanager.a on smithi052 2024-03-26T02:10:44.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:44 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/3407891642' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T02:10:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:10:44.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: Deploying daemon alertmanager.a on smithi052 2024-03-26T02:10:44.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:44 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/3407891642' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T02:10:45.136 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738391 got 34359738391 for osd.0 2024-03-26T02:10:45.137 DEBUG:teuthology.parallel:result is None 2024-03-26T02:10:46.133 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:46.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:46 smithi052 ceph-mon[25889]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T02:10:46.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:46 smithi165 ceph-mon[27627]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T02:10:46.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:46 smithi191 ceph-mon[27701]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T02:10:47.202 INFO:teuthology.orchestra.run.smithi052.stdout:77309411341 2024-03-26T02:10:47.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:47 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:47.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:47 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/1283640921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T02:10:47.710 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:47 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ::ffff:172.21.15.165 - - [26/Mar/2024:02:10:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:10:47.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:47 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:47.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:47 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/1283640921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T02:10:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:47 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:47.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:47 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/1283640921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T02:10:48.594 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411340 got 77309411341 for osd.2 2024-03-26T02:10:48.594 DEBUG:teuthology.parallel:result is None 2024-03-26T02:10:48.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:48 smithi165 ceph-mon[27627]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T02:10:48.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:48 smithi052 ceph-mon[25889]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T02:10:48.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:48 smithi191 ceph-mon[27701]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T02:10:49.526 INFO:teuthology.orchestra.run.smithi052.stdout:55834574868 2024-03-26T02:10:50.382 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574866 got 55834574868 for osd.1 2024-03-26T02:10:50.383 DEBUG:teuthology.parallel:result is None 2024-03-26T02:10:50.383 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T02:10:50.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T02:10:50.632 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:50.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:50 smithi165 ceph-mon[27627]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T02:10:50.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:50 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/726535711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T02:10:50.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:50 smithi052 ceph-mon[25889]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T02:10:50.710 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:50 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/726535711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T02:10:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:50 smithi191 ceph-mon[27701]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T02:10:50.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:50 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/726535711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T02:10:52.595 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:52 smithi052 ceph-mon[25889]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:52.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:52 smithi165 ceph-mon[27627]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:52.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:52 smithi191 ceph-mon[27701]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:53.952 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:10:53.954 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-03-26T02:10:54.556 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:54 smithi052 ceph-mon[25889]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:54.557 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:54 smithi052 ceph-mon[25889]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:10:54.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:54 smithi165 ceph-mon[27627]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:54.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:54 smithi165 ceph-mon[27627]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:10:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:54 smithi191 ceph-mon[27701]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:54.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:54 smithi191 ceph-mon[27701]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:10:55.840 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-03-26T02:10:53.356108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96612,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126556,"statfs":{"total":287972524032,"available":287873593344,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002606"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343751+0000","last_change":"2024-03-26T02:10:09.178848+0000","last_active":"2024-03-26T02:10:31.343751+0000","last_peered":"2024-03-26T02:10:31.343751+0000","last_clean":"2024-03-26T02:10:31.343751+0000","last_became_active":"2024-03-26T02:10:09.178491+0000","last_became_peered":"2024-03-26T02:10:09.178491+0000","last_unstale":"2024-03-26T02:10:31.343751+0000","last_undegraded":"2024-03-26T02:10:31.343751+0000","last_fullsized":"2024-03-26T02:10:31.343751+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:48:02.420816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345389+0000","last_change":"2024-03-26T02:10:07.162573+0000","last_active":"2024-03-26T02:10:31.345389+0000","last_peered":"2024-03-26T02:10:31.345389+0000","last_clean":"2024-03-26T02:10:31.345389+0000","last_became_active":"2024-03-26T02:10:07.161663+0000","last_became_peered":"2024-03-26T02:10:07.161663+0000","last_unstale":"2024-03-26T02:10:31.345389+0000","last_undegraded":"2024-03-26T02:10:31.345389+0000","last_fullsized":"2024-03-26T02:10:31.345389+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:58:58.822836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345612+0000","last_change":"2024-03-26T02:10:03.145978+0000","last_active":"2024-03-26T02:10:31.345612+0000","last_peered":"2024-03-26T02:10:31.345612+0000","last_clean":"2024-03-26T02:10:31.345612+0000","last_became_active":"2024-03-26T02:10:03.145444+0000","last_became_peered":"2024-03-26T02:10:03.145444+0000","last_unstale":"2024-03-26T02:10:31.345612+0000","last_undegraded":"2024-03-26T02:10:31.345612+0000","last_fullsized":"2024-03-26T02:10:31.345612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:53:01.171823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344587+0000","last_change":"2024-03-26T02:10:05.152356+0000","last_active":"2024-03-26T02:10:31.344587+0000","last_peered":"2024-03-26T02:10:31.344587+0000","last_clean":"2024-03-26T02:10:31.344587+0000","last_became_active":"2024-03-26T02:10:05.151925+0000","last_became_peered":"2024-03-26T02:10:05.151925+0000","last_unstale":"2024-03-26T02:10:31.344587+0000","last_undegraded":"2024-03-26T02:10:31.344587+0000","last_fullsized":"2024-03-26T02:10:31.344587+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:45:46.319407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342287+0000","last_change":"2024-03-26T02:10:09.172611+0000","last_active":"2024-03-26T02:10:31.342287+0000","last_peered":"2024-03-26T02:10:31.342287+0000","last_clean":"2024-03-26T02:10:31.342287+0000","last_became_active":"2024-03-26T02:10:09.172327+0000","last_became_peered":"2024-03-26T02:10:09.172327+0000","last_unstale":"2024-03-26T02:10:31.342287+0000","last_undegraded":"2024-03-26T02:10:31.342287+0000","last_fullsized":"2024-03-26T02:10:31.342287+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:26:24.560661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344225+0000","last_change":"2024-03-26T02:10:07.158999+0000","last_active":"2024-03-26T02:10:31.344225+0000","last_peered":"2024-03-26T02:10:31.344225+0000","last_clean":"2024-03-26T02:10:31.344225+0000","last_became_active":"2024-03-26T02:10:07.157751+0000","last_became_peered":"2024-03-26T02:10:07.157751+0000","last_unstale":"2024-03-26T02:10:31.344225+0000","last_undegraded":"2024-03-26T02:10:31.344225+0000","last_fullsized":"2024-03-26T02:10:31.344225+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:48:48.875583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345072+0000","last_change":"2024-03-26T02:10:03.146351+0000","last_active":"2024-03-26T02:10:31.345072+0000","last_peered":"2024-03-26T02:10:31.345072+0000","last_clean":"2024-03-26T02:10:31.345072+0000","last_became_active":"2024-03-26T02:10:03.145242+0000","last_became_peered":"2024-03-26T02:10:03.145242+0000","last_unstale":"2024-03-26T02:10:31.345072+0000","last_undegraded":"2024-03-26T02:10:31.345072+0000","last_fullsized":"2024-03-26T02:10:31.345072+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:32:18.434819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345555+0000","last_change":"2024-03-26T02:10:05.151212+0000","last_active":"2024-03-26T02:10:31.345555+0000","last_peered":"2024-03-26T02:10:31.345555+0000","last_clean":"2024-03-26T02:10:31.345555+0000","last_became_active":"2024-03-26T02:10:05.150844+0000","last_became_peered":"2024-03-26T02:10:05.150844+0000","last_unstale":"2024-03-26T02:10:31.345555+0000","last_undegraded":"2024-03-26T02:10:31.345555+0000","last_fullsized":"2024-03-26T02:10:31.345555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:47:21.893622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346808+0000","last_change":"2024-03-26T02:10:09.171246+0000","last_active":"2024-03-26T02:10:31.346808+0000","last_peered":"2024-03-26T02:10:31.346808+0000","last_clean":"2024-03-26T02:10:31.346808+0000","last_became_active":"2024-03-26T02:10:09.170781+0000","last_became_peered":"2024-03-26T02:10:09.170781+0000","last_unstale":"2024-03-26T02:10:31.346808+0000","last_undegraded":"2024-03-26T02:10:31.346808+0000","last_fullsized":"2024-03-26T02:10:31.346808+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:41:00.578858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344327+0000","last_change":"2024-03-26T02:10:07.160672+0000","last_active":"2024-03-26T02:10:31.344327+0000","last_peered":"2024-03-26T02:10:31.344327+0000","last_clean":"2024-03-26T02:10:31.344327+0000","last_became_active":"2024-03-26T02:10:07.159880+0000","last_became_peered":"2024-03-26T02:10:07.159880+0000","last_unstale":"2024-03-26T02:10:31.344327+0000","last_undegraded":"2024-03-26T02:10:31.344327+0000","last_fullsized":"2024-03-26T02:10:31.344327+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:10:46.621557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348597+0000","last_change":"2024-03-26T02:10:03.149069+0000","last_active":"2024-03-26T02:10:31.348597+0000","last_peered":"2024-03-26T02:10:31.348597+0000","last_clean":"2024-03-26T02:10:31.348597+0000","last_became_active":"2024-03-26T02:10:03.147674+0000","last_became_peered":"2024-03-26T02:10:03.147674+0000","last_unstale":"2024-03-26T02:10:31.348597+0000","last_undegraded":"2024-03-26T02:10:31.348597+0000","last_fullsized":"2024-03-26T02:10:31.348597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:42:21.413893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346930+0000","last_change":"2024-03-26T02:10:05.151500+0000","last_active":"2024-03-26T02:10:31.346930+0000","last_peered":"2024-03-26T02:10:31.346930+0000","last_clean":"2024-03-26T02:10:31.346930+0000","last_became_active":"2024-03-26T02:10:05.151163+0000","last_became_peered":"2024-03-26T02:10:05.151163+0000","last_unstale":"2024-03-26T02:10:31.346930+0000","last_undegraded":"2024-03-26T02:10:31.346930+0000","last_fullsized":"2024-03-26T02:10:31.346930+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:22.146610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344043+0000","last_change":"2024-03-26T02:10:09.180164+0000","last_active":"2024-03-26T02:10:31.344043+0000","last_peered":"2024-03-26T02:10:31.344043+0000","last_clean":"2024-03-26T02:10:31.344043+0000","last_became_active":"2024-03-26T02:10:09.179725+0000","last_became_peered":"2024-03-26T02:10:09.179725+0000","last_unstale":"2024-03-26T02:10:31.344043+0000","last_undegraded":"2024-03-26T02:10:31.344043+0000","last_fullsized":"2024-03-26T02:10:31.344043+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:10:42.248845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344669+0000","last_change":"2024-03-26T02:10:07.158187+0000","last_active":"2024-03-26T02:10:31.344669+0000","last_peered":"2024-03-26T02:10:31.344669+0000","last_clean":"2024-03-26T02:10:31.344669+0000","last_became_active":"2024-03-26T02:10:07.157653+0000","last_became_peered":"2024-03-26T02:10:07.157653+0000","last_unstale":"2024-03-26T02:10:31.344669+0000","last_undegraded":"2024-03-26T02:10:31.344669+0000","last_fullsized":"2024-03-26T02:10:31.344669+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:45:43.729359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347361+0000","last_change":"2024-03-26T02:10:03.145736+0000","last_active":"2024-03-26T02:10:31.347361+0000","last_peered":"2024-03-26T02:10:31.347361+0000","last_clean":"2024-03-26T02:10:31.347361+0000","last_became_active":"2024-03-26T02:10:03.145404+0000","last_became_peered":"2024-03-26T02:10:03.145404+0000","last_unstale":"2024-03-26T02:10:31.347361+0000","last_undegraded":"2024-03-26T02:10:31.347361+0000","last_fullsized":"2024-03-26T02:10:31.347361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:04:18.823386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'18","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348466+0000","last_change":"2024-03-26T02:10:05.149031+0000","last_active":"2024-03-26T02:10:31.348466+0000","last_peered":"2024-03-26T02:10:31.348466+0000","last_clean":"2024-03-26T02:10:31.348466+0000","last_became_active":"2024-03-26T02:10:05.148447+0000","last_became_peered":"2024-03-26T02:10:05.148447+0000","last_unstale":"2024-03-26T02:10:31.348466+0000","last_undegraded":"2024-03-26T02:10:31.348466+0000","last_fullsized":"2024-03-26T02:10:31.348466+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:37:38.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345581+0000","last_change":"2024-03-26T02:10:09.180017+0000","last_active":"2024-03-26T02:10:31.345581+0000","last_peered":"2024-03-26T02:10:31.345581+0000","last_clean":"2024-03-26T02:10:31.345581+0000","last_became_active":"2024-03-26T02:10:09.179644+0000","last_became_peered":"2024-03-26T02:10:09.179644+0000","last_unstale":"2024-03-26T02:10:31.345581+0000","last_undegraded":"2024-03-26T02:10:31.345581+0000","last_fullsized":"2024-03-26T02:10:31.345581+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:20:16.575058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348078+0000","last_change":"2024-03-26T02:10:07.162457+0000","last_active":"2024-03-26T02:10:31.348078+0000","last_peered":"2024-03-26T02:10:31.348078+0000","last_clean":"2024-03-26T02:10:31.348078+0000","last_became_active":"2024-03-26T02:10:07.161593+0000","last_became_peered":"2024-03-26T02:10:07.161593+0000","last_unstale":"2024-03-26T02:10:31.348078+0000","last_undegraded":"2024-03-26T02:10:31.348078+0000","last_fullsized":"2024-03-26T02:10:31.348078+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:08:49.788507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346178+0000","last_change":"2024-03-26T02:10:03.147724+0000","last_active":"2024-03-26T02:10:31.346178+0000","last_peered":"2024-03-26T02:10:31.346178+0000","last_clean":"2024-03-26T02:10:31.346178+0000","last_became_active":"2024-03-26T02:10:03.145771+0000","last_became_peered":"2024-03-26T02:10:03.145771+0000","last_unstale":"2024-03-26T02:10:31.346178+0000","last_undegraded":"2024-03-26T02:10:31.346178+0000","last_fullsized":"2024-03-26T02:10:31.346178+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:16:33.102380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'22","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348229+0000","last_change":"2024-03-26T02:10:05.147593+0000","last_active":"2024-03-26T02:10:31.348229+0000","last_peered":"2024-03-26T02:10:31.348229+0000","last_clean":"2024-03-26T02:10:31.348229+0000","last_became_active":"2024-03-26T02:10:05.147292+0000","last_became_peered":"2024-03-26T02:10:05.147292+0000","last_unstale":"2024-03-26T02:10:31.348229+0000","last_undegraded":"2024-03-26T02:10:31.348229+0000","last_fullsized":"2024-03-26T02:10:31.348229+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:22:28.271533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343780+0000","last_change":"2024-03-26T02:10:09.174779+0000","last_active":"2024-03-26T02:10:31.343780+0000","last_peered":"2024-03-26T02:10:31.343780+0000","last_clean":"2024-03-26T02:10:31.343780+0000","last_became_active":"2024-03-26T02:10:09.174433+0000","last_became_peered":"2024-03-26T02:10:09.174433+0000","last_unstale":"2024-03-26T02:10:31.343780+0000","last_undegraded":"2024-03-26T02:10:31.343780+0000","last_fullsized":"2024-03-26T02:10:31.343780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:15:05.523966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347174+0000","last_change":"2024-03-26T02:10:07.162758+0000","last_active":"2024-03-26T02:10:31.347174+0000","last_peered":"2024-03-26T02:10:31.347174+0000","last_clean":"2024-03-26T02:10:31.347174+0000","last_became_active":"2024-03-26T02:10:07.161766+0000","last_became_peered":"2024-03-26T02:10:07.161766+0000","last_unstale":"2024-03-26T02:10:31.347174+0000","last_undegraded":"2024-03-26T02:10:31.347174+0000","last_fullsized":"2024-03-26T02:10:31.347174+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:14:59.629464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345198+0000","last_change":"2024-03-26T02:10:03.146476+0000","last_active":"2024-03-26T02:10:31.345198+0000","last_peered":"2024-03-26T02:10:31.345198+0000","last_clean":"2024-03-26T02:10:31.345198+0000","last_became_active":"2024-03-26T02:10:03.145883+0000","last_became_peered":"2024-03-26T02:10:03.145883+0000","last_unstale":"2024-03-26T02:10:31.345198+0000","last_undegraded":"2024-03-26T02:10:31.345198+0000","last_fullsized":"2024-03-26T02:10:31.345198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:28:25.046266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346031+0000","last_change":"2024-03-26T02:10:05.148750+0000","last_active":"2024-03-26T02:10:31.346031+0000","last_peered":"2024-03-26T02:10:31.346031+0000","last_clean":"2024-03-26T02:10:31.346031+0000","last_became_active":"2024-03-26T02:10:05.148267+0000","last_became_peered":"2024-03-26T02:10:05.148267+0000","last_unstale":"2024-03-26T02:10:31.346031+0000","last_undegraded":"2024-03-26T02:10:31.346031+0000","last_fullsized":"2024-03-26T02:10:31.346031+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:57:14.794777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344379+0000","last_change":"2024-03-26T02:10:09.176305+0000","last_active":"2024-03-26T02:10:31.344379+0000","last_peered":"2024-03-26T02:10:31.344379+0000","last_clean":"2024-03-26T02:10:31.344379+0000","last_became_active":"2024-03-26T02:10:09.175277+0000","last_became_peered":"2024-03-26T02:10:09.175277+0000","last_unstale":"2024-03-26T02:10:31.344379+0000","last_undegraded":"2024-03-26T02:10:31.344379+0000","last_fullsized":"2024-03-26T02:10:31.344379+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:16:52.365975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:43.197437+0000","last_change":"2024-03-26T02:10:07.165428+0000","last_active":"2024-03-26T02:10:43.197437+0000","last_peered":"2024-03-26T02:10:43.197437+0000","last_clean":"2024-03-26T02:10:43.197437+0000","last_became_active":"2024-03-26T02:10:07.164967+0000","last_became_peered":"2024-03-26T02:10:07.164967+0000","last_unstale":"2024-03-26T02:10:43.197437+0000","last_undegraded":"2024-03-26T02:10:43.197437+0000","last_fullsized":"2024-03-26T02:10:43.197437+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:17:30.903932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344719+0000","last_change":"2024-03-26T02:10:03.146758+0000","last_active":"2024-03-26T02:10:31.344719+0000","last_peered":"2024-03-26T02:10:31.344719+0000","last_clean":"2024-03-26T02:10:31.344719+0000","last_became_active":"2024-03-26T02:10:03.145491+0000","last_became_peered":"2024-03-26T02:10:03.145491+0000","last_unstale":"2024-03-26T02:10:31.344719+0000","last_undegraded":"2024-03-26T02:10:31.344719+0000","last_fullsized":"2024-03-26T02:10:31.344719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:04:08.395675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342019+0000","last_change":"2024-03-26T02:10:05.151580+0000","last_active":"2024-03-26T02:10:31.342019+0000","last_peered":"2024-03-26T02:10:31.342019+0000","last_clean":"2024-03-26T02:10:31.342019+0000","last_became_active":"2024-03-26T02:10:05.151179+0000","last_became_peered":"2024-03-26T02:10:05.151179+0000","last_unstale":"2024-03-26T02:10:31.342019+0000","last_undegraded":"2024-03-26T02:10:31.342019+0000","last_fullsized":"2024-03-26T02:10:31.342019+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:11:16.297115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344660+0000","last_change":"2024-03-26T02:10:09.178215+0000","last_active":"2024-03-26T02:10:31.344660+0000","last_peered":"2024-03-26T02:10:31.344660+0000","last_clean":"2024-03-26T02:10:31.344660+0000","last_became_active":"2024-03-26T02:10:09.177638+0000","last_became_peered":"2024-03-26T02:10:09.177638+0000","last_unstale":"2024-03-26T02:10:31.344660+0000","last_undegraded":"2024-03-26T02:10:31.344660+0000","last_fullsized":"2024-03-26T02:10:31.344660+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:47:18.325957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344444+0000","last_change":"2024-03-26T02:10:07.156833+0000","last_active":"2024-03-26T02:10:31.344444+0000","last_peered":"2024-03-26T02:10:31.344444+0000","last_clean":"2024-03-26T02:10:31.344444+0000","last_became_active":"2024-03-26T02:10:07.156018+0000","last_became_peered":"2024-03-26T02:10:07.156018+0000","last_unstale":"2024-03-26T02:10:31.344444+0000","last_undegraded":"2024-03-26T02:10:31.344444+0000","last_fullsized":"2024-03-26T02:10:31.344444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:29:16.614231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345530+0000","last_change":"2024-03-26T02:10:03.149104+0000","last_active":"2024-03-26T02:10:31.345530+0000","last_peered":"2024-03-26T02:10:31.345530+0000","last_clean":"2024-03-26T02:10:31.345530+0000","last_became_active":"2024-03-26T02:10:03.147860+0000","last_became_peered":"2024-03-26T02:10:03.147860+0000","last_unstale":"2024-03-26T02:10:31.345530+0000","last_undegraded":"2024-03-26T02:10:31.345530+0000","last_fullsized":"2024-03-26T02:10:31.345530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:13:48.500636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'39","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342020+0000","last_change":"2024-03-26T02:10:05.150313+0000","last_active":"2024-03-26T02:10:31.342020+0000","last_peered":"2024-03-26T02:10:31.342020+0000","last_clean":"2024-03-26T02:10:31.342020+0000","last_became_active":"2024-03-26T02:10:05.149889+0000","last_became_peered":"2024-03-26T02:10:05.149889+0000","last_unstale":"2024-03-26T02:10:31.342020+0000","last_undegraded":"2024-03-26T02:10:31.342020+0000","last_fullsized":"2024-03-26T02:10:31.342020+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:49:15.417727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344510+0000","last_change":"2024-03-26T02:10:09.180826+0000","last_active":"2024-03-26T02:10:31.344510+0000","last_peered":"2024-03-26T02:10:31.344510+0000","last_clean":"2024-03-26T02:10:31.344510+0000","last_became_active":"2024-03-26T02:10:09.180484+0000","last_became_peered":"2024-03-26T02:10:09.180484+0000","last_unstale":"2024-03-26T02:10:31.344510+0000","last_undegraded":"2024-03-26T02:10:31.344510+0000","last_fullsized":"2024-03-26T02:10:31.344510+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:08:35.589827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.341894+0000","last_change":"2024-03-26T02:10:07.155543+0000","last_active":"2024-03-26T02:10:31.341894+0000","last_peered":"2024-03-26T02:10:31.341894+0000","last_clean":"2024-03-26T02:10:31.341894+0000","last_became_active":"2024-03-26T02:10:07.155227+0000","last_became_peered":"2024-03-26T02:10:07.155227+0000","last_unstale":"2024-03-26T02:10:31.341894+0000","last_undegraded":"2024-03-26T02:10:31.341894+0000","last_fullsized":"2024-03-26T02:10:31.341894+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:11:33.780128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345918+0000","last_change":"2024-03-26T02:10:03.148045+0000","last_active":"2024-03-26T02:10:31.345918+0000","last_peered":"2024-03-26T02:10:31.345918+0000","last_clean":"2024-03-26T02:10:31.345918+0000","last_became_active":"2024-03-26T02:10:03.147108+0000","last_became_peered":"2024-03-26T02:10:03.147108+0000","last_unstale":"2024-03-26T02:10:31.345918+0000","last_undegraded":"2024-03-26T02:10:31.345918+0000","last_fullsized":"2024-03-26T02:10:31.345918+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:45:15.645475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345283+0000","last_change":"2024-03-26T02:10:05.149174+0000","last_active":"2024-03-26T02:10:31.345283+0000","last_peered":"2024-03-26T02:10:31.345283+0000","last_clean":"2024-03-26T02:10:31.345283+0000","last_became_active":"2024-03-26T02:10:05.148232+0000","last_became_peered":"2024-03-26T02:10:05.148232+0000","last_unstale":"2024-03-26T02:10:31.345283+0000","last_undegraded":"2024-03-26T02:10:31.345283+0000","last_fullsized":"2024-03-26T02:10:31.345283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:06:07.747446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345632+0000","last_change":"2024-03-26T02:10:09.176384+0000","last_active":"2024-03-26T02:10:31.345632+0000","last_peered":"2024-03-26T02:10:31.345632+0000","last_clean":"2024-03-26T02:10:31.345632+0000","last_became_active":"2024-03-26T02:10:09.175965+0000","last_became_peered":"2024-03-26T02:10:09.175965+0000","last_unstale":"2024-03-26T02:10:31.345632+0000","last_undegraded":"2024-03-26T02:10:31.345632+0000","last_fullsized":"2024-03-26T02:10:31.345632+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:50:24.176240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342319+0000","last_change":"2024-03-26T02:10:07.157393+0000","last_active":"2024-03-26T02:10:31.342319+0000","last_peered":"2024-03-26T02:10:31.342319+0000","last_clean":"2024-03-26T02:10:31.342319+0000","last_became_active":"2024-03-26T02:10:07.156932+0000","last_became_peered":"2024-03-26T02:10:07.156932+0000","last_unstale":"2024-03-26T02:10:31.342319+0000","last_undegraded":"2024-03-26T02:10:31.342319+0000","last_fullsized":"2024-03-26T02:10:31.342319+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:44:28.689508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345023+0000","last_change":"2024-03-26T02:10:03.150694+0000","last_active":"2024-03-26T02:10:31.345023+0000","last_peered":"2024-03-26T02:10:31.345023+0000","last_clean":"2024-03-26T02:10:31.345023+0000","last_became_active":"2024-03-26T02:10:03.150121+0000","last_became_peered":"2024-03-26T02:10:03.150121+0000","last_unstale":"2024-03-26T02:10:31.345023+0000","last_undegraded":"2024-03-26T02:10:31.345023+0000","last_fullsized":"2024-03-26T02:10:31.345023+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:11:17.951007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345765+0000","last_change":"2024-03-26T02:10:05.151468+0000","last_active":"2024-03-26T02:10:31.345765+0000","last_peered":"2024-03-26T02:10:31.345765+0000","last_clean":"2024-03-26T02:10:31.345765+0000","last_became_active":"2024-03-26T02:10:05.151035+0000","last_became_peered":"2024-03-26T02:10:05.151035+0000","last_unstale":"2024-03-26T02:10:31.345765+0000","last_undegraded":"2024-03-26T02:10:31.345765+0000","last_fullsized":"2024-03-26T02:10:31.345765+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:38:43.396252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346963+0000","last_change":"2024-03-26T02:10:09.171573+0000","last_active":"2024-03-26T02:10:31.346963+0000","last_peered":"2024-03-26T02:10:31.346963+0000","last_clean":"2024-03-26T02:10:31.346963+0000","last_became_active":"2024-03-26T02:10:09.171283+0000","last_became_peered":"2024-03-26T02:10:09.171283+0000","last_unstale":"2024-03-26T02:10:31.346963+0000","last_undegraded":"2024-03-26T02:10:31.346963+0000","last_fullsized":"2024-03-26T02:10:31.346963+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:50:36.130678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344505+0000","last_change":"2024-03-26T02:10:07.159762+0000","last_active":"2024-03-26T02:10:31.344505+0000","last_peered":"2024-03-26T02:10:31.344505+0000","last_clean":"2024-03-26T02:10:31.344505+0000","last_became_active":"2024-03-26T02:10:07.159344+0000","last_became_peered":"2024-03-26T02:10:07.159344+0000","last_unstale":"2024-03-26T02:10:31.344505+0000","last_undegraded":"2024-03-26T02:10:31.344505+0000","last_fullsized":"2024-03-26T02:10:31.344505+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:14:00.267421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348457+0000","last_change":"2024-03-26T02:10:03.147577+0000","last_active":"2024-03-26T02:10:31.348457+0000","last_peered":"2024-03-26T02:10:31.348457+0000","last_clean":"2024-03-26T02:10:31.348457+0000","last_became_active":"2024-03-26T02:10:03.146249+0000","last_became_peered":"2024-03-26T02:10:03.146249+0000","last_unstale":"2024-03-26T02:10:31.348457+0000","last_undegraded":"2024-03-26T02:10:31.348457+0000","last_fullsized":"2024-03-26T02:10:31.348457+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:15:18.236497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346893+0000","last_change":"2024-03-26T02:10:05.149586+0000","last_active":"2024-03-26T02:10:31.346893+0000","last_peered":"2024-03-26T02:10:31.346893+0000","last_clean":"2024-03-26T02:10:31.346893+0000","last_became_active":"2024-03-26T02:10:05.148951+0000","last_became_peered":"2024-03-26T02:10:05.148951+0000","last_unstale":"2024-03-26T02:10:31.346893+0000","last_undegraded":"2024-03-26T02:10:31.346893+0000","last_fullsized":"2024-03-26T02:10:31.346893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:25:42.506433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345597+0000","last_change":"2024-03-26T02:10:09.178976+0000","last_active":"2024-03-26T02:10:31.345597+0000","last_peered":"2024-03-26T02:10:31.345597+0000","last_clean":"2024-03-26T02:10:31.345597+0000","last_became_active":"2024-03-26T02:10:09.178618+0000","last_became_peered":"2024-03-26T02:10:09.178618+0000","last_unstale":"2024-03-26T02:10:31.345597+0000","last_undegraded":"2024-03-26T02:10:31.345597+0000","last_fullsized":"2024-03-26T02:10:31.345597+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:52:13.605859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342190+0000","last_change":"2024-03-26T02:10:07.156494+0000","last_active":"2024-03-26T02:10:31.342190+0000","last_peered":"2024-03-26T02:10:31.342190+0000","last_clean":"2024-03-26T02:10:31.342190+0000","last_became_active":"2024-03-26T02:10:07.156153+0000","last_became_peered":"2024-03-26T02:10:07.156153+0000","last_unstale":"2024-03-26T02:10:31.342190+0000","last_undegraded":"2024-03-26T02:10:31.342190+0000","last_fullsized":"2024-03-26T02:10:31.342190+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:21:42.466813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346099+0000","last_change":"2024-03-26T02:10:03.150380+0000","last_active":"2024-03-26T02:10:31.346099+0000","last_peered":"2024-03-26T02:10:31.346099+0000","last_clean":"2024-03-26T02:10:31.346099+0000","last_became_active":"2024-03-26T02:10:03.150047+0000","last_became_peered":"2024-03-26T02:10:03.150047+0000","last_unstale":"2024-03-26T02:10:31.346099+0000","last_undegraded":"2024-03-26T02:10:31.346099+0000","last_fullsized":"2024-03-26T02:10:31.346099+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:47:40.410519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348279+0000","last_change":"2024-03-26T02:10:05.148761+0000","last_active":"2024-03-26T02:10:31.348279+0000","last_peered":"2024-03-26T02:10:31.348279+0000","last_clean":"2024-03-26T02:10:31.348279+0000","last_became_active":"2024-03-26T02:10:05.148503+0000","last_became_peered":"2024-03-26T02:10:05.148503+0000","last_unstale":"2024-03-26T02:10:31.348279+0000","last_undegraded":"2024-03-26T02:10:31.348279+0000","last_fullsized":"2024-03-26T02:10:31.348279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:28:31.258420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343511+0000","last_change":"2024-03-26T02:10:09.176185+0000","last_active":"2024-03-26T02:10:31.343511+0000","last_peered":"2024-03-26T02:10:31.343511+0000","last_clean":"2024-03-26T02:10:31.343511+0000","last_became_active":"2024-03-26T02:10:09.174980+0000","last_became_peered":"2024-03-26T02:10:09.174980+0000","last_unstale":"2024-03-26T02:10:31.343511+0000","last_undegraded":"2024-03-26T02:10:31.343511+0000","last_fullsized":"2024-03-26T02:10:31.343511+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:42:54.816228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:43.197440+0000","last_change":"2024-03-26T02:10:07.163792+0000","last_active":"2024-03-26T02:10:43.197440+0000","last_peered":"2024-03-26T02:10:43.197440+0000","last_clean":"2024-03-26T02:10:43.197440+0000","last_became_active":"2024-03-26T02:10:07.163431+0000","last_became_peered":"2024-03-26T02:10:07.163431+0000","last_unstale":"2024-03-26T02:10:43.197440+0000","last_undegraded":"2024-03-26T02:10:43.197440+0000","last_fullsized":"2024-03-26T02:10:43.197440+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:53:46.495755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345936+0000","last_change":"2024-03-26T02:10:03.148408+0000","last_active":"2024-03-26T02:10:31.345936+0000","last_peered":"2024-03-26T02:10:31.345936+0000","last_clean":"2024-03-26T02:10:31.345936+0000","last_became_active":"2024-03-26T02:10:03.147567+0000","last_became_peered":"2024-03-26T02:10:03.147567+0000","last_unstale":"2024-03-26T02:10:31.345936+0000","last_undegraded":"2024-03-26T02:10:31.345936+0000","last_fullsized":"2024-03-26T02:10:31.345936+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:37:57.113288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342450+0000","last_change":"2024-03-26T02:10:05.150995+0000","last_active":"2024-03-26T02:10:31.342450+0000","last_peered":"2024-03-26T02:10:31.342450+0000","last_clean":"2024-03-26T02:10:31.342450+0000","last_became_active":"2024-03-26T02:10:05.150551+0000","last_became_peered":"2024-03-26T02:10:05.150551+0000","last_unstale":"2024-03-26T02:10:31.342450+0000","last_undegraded":"2024-03-26T02:10:31.342450+0000","last_fullsized":"2024-03-26T02:10:31.342450+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:10:22.839667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345753+0000","last_change":"2024-03-26T02:10:09.180546+0000","last_active":"2024-03-26T02:10:31.345753+0000","last_peered":"2024-03-26T02:10:31.345753+0000","last_clean":"2024-03-26T02:10:31.345753+0000","last_became_active":"2024-03-26T02:10:09.180012+0000","last_became_peered":"2024-03-26T02:10:09.180012+0000","last_unstale":"2024-03-26T02:10:31.345753+0000","last_undegraded":"2024-03-26T02:10:31.345753+0000","last_fullsized":"2024-03-26T02:10:31.345753+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:54:37.899012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:43.197410+0000","last_change":"2024-03-26T02:10:07.162776+0000","last_active":"2024-03-26T02:10:43.197410+0000","last_peered":"2024-03-26T02:10:43.197410+0000","last_clean":"2024-03-26T02:10:43.197410+0000","last_became_active":"2024-03-26T02:10:07.162329+0000","last_became_peered":"2024-03-26T02:10:07.162329+0000","last_unstale":"2024-03-26T02:10:43.197410+0000","last_undegraded":"2024-03-26T02:10:43.197410+0000","last_fullsized":"2024-03-26T02:10:43.197410+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:15:46.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347861+0000","last_change":"2024-03-26T02:10:03.147290+0000","last_active":"2024-03-26T02:10:31.347861+0000","last_peered":"2024-03-26T02:10:31.347861+0000","last_clean":"2024-03-26T02:10:31.347861+0000","last_became_active":"2024-03-26T02:10:03.145638+0000","last_became_peered":"2024-03-26T02:10:03.145638+0000","last_unstale":"2024-03-26T02:10:31.347861+0000","last_undegraded":"2024-03-26T02:10:31.347861+0000","last_fullsized":"2024-03-26T02:10:31.347861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:14:04.616380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345898+0000","last_change":"2024-03-26T02:10:05.153923+0000","last_active":"2024-03-26T02:10:31.345898+0000","last_peered":"2024-03-26T02:10:31.345898+0000","last_clean":"2024-03-26T02:10:31.345898+0000","last_became_active":"2024-03-26T02:10:05.153340+0000","last_became_peered":"2024-03-26T02:10:05.153340+0000","last_unstale":"2024-03-26T02:10:31.345898+0000","last_undegraded":"2024-03-26T02:10:31.345898+0000","last_fullsized":"2024-03-26T02:10:31.345898+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:16:22.378934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344324+0000","last_change":"2024-03-26T02:10:09.177959+0000","last_active":"2024-03-26T02:10:31.344324+0000","last_peered":"2024-03-26T02:10:31.344324+0000","last_clean":"2024-03-26T02:10:31.344324+0000","last_became_active":"2024-03-26T02:10:09.177595+0000","last_became_peered":"2024-03-26T02:10:09.177595+0000","last_unstale":"2024-03-26T02:10:31.344324+0000","last_undegraded":"2024-03-26T02:10:31.344324+0000","last_fullsized":"2024-03-26T02:10:31.344324+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:24:03.848583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345387+0000","last_change":"2024-03-26T02:10:07.157617+0000","last_active":"2024-03-26T02:10:31.345387+0000","last_peered":"2024-03-26T02:10:31.345387+0000","last_clean":"2024-03-26T02:10:31.345387+0000","last_became_active":"2024-03-26T02:10:07.156492+0000","last_became_peered":"2024-03-26T02:10:07.156492+0000","last_unstale":"2024-03-26T02:10:31.345387+0000","last_undegraded":"2024-03-26T02:10:31.345387+0000","last_fullsized":"2024-03-26T02:10:31.345387+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:33:15.838472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344686+0000","last_change":"2024-03-26T02:10:03.151316+0000","last_active":"2024-03-26T02:10:31.344686+0000","last_peered":"2024-03-26T02:10:31.344686+0000","last_clean":"2024-03-26T02:10:31.344686+0000","last_became_active":"2024-03-26T02:10:03.150852+0000","last_became_peered":"2024-03-26T02:10:03.150852+0000","last_unstale":"2024-03-26T02:10:31.344686+0000","last_undegraded":"2024-03-26T02:10:31.344686+0000","last_fullsized":"2024-03-26T02:10:31.344686+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:26:17.723569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343755+0000","last_change":"2024-03-26T02:10:05.152169+0000","last_active":"2024-03-26T02:10:31.343755+0000","last_peered":"2024-03-26T02:10:31.343755+0000","last_clean":"2024-03-26T02:10:31.343755+0000","last_became_active":"2024-03-26T02:10:05.151896+0000","last_became_peered":"2024-03-26T02:10:05.151896+0000","last_unstale":"2024-03-26T02:10:31.343755+0000","last_undegraded":"2024-03-26T02:10:31.343755+0000","last_fullsized":"2024-03-26T02:10:31.343755+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:23:34.403599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345130+0000","last_change":"2024-03-26T02:10:09.172840+0000","last_active":"2024-03-26T02:10:31.345130+0000","last_peered":"2024-03-26T02:10:31.345130+0000","last_clean":"2024-03-26T02:10:31.345130+0000","last_became_active":"2024-03-26T02:10:09.172503+0000","last_became_peered":"2024-03-26T02:10:09.172503+0000","last_unstale":"2024-03-26T02:10:31.345130+0000","last_undegraded":"2024-03-26T02:10:31.345130+0000","last_fullsized":"2024-03-26T02:10:31.345130+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:13:27.971276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344483+0000","last_change":"2024-03-26T02:10:07.157545+0000","last_active":"2024-03-26T02:10:31.344483+0000","last_peered":"2024-03-26T02:10:31.344483+0000","last_clean":"2024-03-26T02:10:31.344483+0000","last_became_active":"2024-03-26T02:10:07.157098+0000","last_became_peered":"2024-03-26T02:10:07.157098+0000","last_unstale":"2024-03-26T02:10:31.344483+0000","last_undegraded":"2024-03-26T02:10:31.344483+0000","last_fullsized":"2024-03-26T02:10:31.344483+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:51:51.549880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345869+0000","last_change":"2024-03-26T02:10:03.149259+0000","last_active":"2024-03-26T02:10:31.345869+0000","last_peered":"2024-03-26T02:10:31.345869+0000","last_clean":"2024-03-26T02:10:31.345869+0000","last_became_active":"2024-03-26T02:10:03.148528+0000","last_became_peered":"2024-03-26T02:10:03.148528+0000","last_unstale":"2024-03-26T02:10:31.345869+0000","last_undegraded":"2024-03-26T02:10:31.345869+0000","last_fullsized":"2024-03-26T02:10:31.345869+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:25.678733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346670+0000","last_change":"2024-03-26T02:10:05.148573+0000","last_active":"2024-03-26T02:10:31.346670+0000","last_peered":"2024-03-26T02:10:31.346670+0000","last_clean":"2024-03-26T02:10:31.346670+0000","last_became_active":"2024-03-26T02:10:05.148213+0000","last_became_peered":"2024-03-26T02:10:05.148213+0000","last_unstale":"2024-03-26T02:10:31.346670+0000","last_undegraded":"2024-03-26T02:10:31.346670+0000","last_fullsized":"2024-03-26T02:10:31.346670+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:21:51.711663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346319+0000","last_change":"2024-03-26T02:10:09.178594+0000","last_active":"2024-03-26T02:10:31.346319+0000","last_peered":"2024-03-26T02:10:31.346319+0000","last_clean":"2024-03-26T02:10:31.346319+0000","last_became_active":"2024-03-26T02:10:09.178342+0000","last_became_peered":"2024-03-26T02:10:09.178342+0000","last_unstale":"2024-03-26T02:10:31.346319+0000","last_undegraded":"2024-03-26T02:10:31.346319+0000","last_fullsized":"2024-03-26T02:10:31.346319+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:00:27.540660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348426+0000","last_change":"2024-03-26T02:10:03.146736+0000","last_active":"2024-03-26T02:10:31.348426+0000","last_peered":"2024-03-26T02:10:31.348426+0000","last_clean":"2024-03-26T02:10:31.348426+0000","last_became_active":"2024-03-26T02:10:03.146064+0000","last_became_peered":"2024-03-26T02:10:03.146064+0000","last_unstale":"2024-03-26T02:10:31.348426+0000","last_undegraded":"2024-03-26T02:10:31.348426+0000","last_fullsized":"2024-03-26T02:10:31.348426+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:56:26.049371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348039+0000","last_change":"2024-03-26T02:10:05.147570+0000","last_active":"2024-03-26T02:10:31.348039+0000","last_peered":"2024-03-26T02:10:31.348039+0000","last_clean":"2024-03-26T02:10:31.348039+0000","last_became_active":"2024-03-26T02:10:05.147225+0000","last_became_peered":"2024-03-26T02:10:05.147225+0000","last_unstale":"2024-03-26T02:10:31.348039+0000","last_undegraded":"2024-03-26T02:10:31.348039+0000","last_fullsized":"2024-03-26T02:10:31.348039+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:32:17.626514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:43.197488+0000","last_change":"2024-03-26T02:10:07.162899+0000","last_active":"2024-03-26T02:10:43.197488+0000","last_peered":"2024-03-26T02:10:43.197488+0000","last_clean":"2024-03-26T02:10:43.197488+0000","last_became_active":"2024-03-26T02:10:07.162496+0000","last_became_peered":"2024-03-26T02:10:07.162496+0000","last_unstale":"2024-03-26T02:10:43.197488+0000","last_undegraded":"2024-03-26T02:10:43.197488+0000","last_fullsized":"2024-03-26T02:10:43.197488+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:50:58.598137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342347+0000","last_change":"2024-03-26T02:10:09.174518+0000","last_active":"2024-03-26T02:10:31.342347+0000","last_peered":"2024-03-26T02:10:31.342347+0000","last_clean":"2024-03-26T02:10:31.342347+0000","last_became_active":"2024-03-26T02:10:09.174133+0000","last_became_peered":"2024-03-26T02:10:09.174133+0000","last_unstale":"2024-03-26T02:10:31.342347+0000","last_undegraded":"2024-03-26T02:10:31.342347+0000","last_fullsized":"2024-03-26T02:10:31.342347+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:28:22.095541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343814+0000","last_change":"2024-03-26T02:10:03.145727+0000","last_active":"2024-03-26T02:10:31.343814+0000","last_peered":"2024-03-26T02:10:31.343814+0000","last_clean":"2024-03-26T02:10:31.343814+0000","last_became_active":"2024-03-26T02:10:03.145184+0000","last_became_peered":"2024-03-26T02:10:03.145184+0000","last_unstale":"2024-03-26T02:10:31.343814+0000","last_undegraded":"2024-03-26T02:10:31.343814+0000","last_fullsized":"2024-03-26T02:10:31.343814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:03:28.975481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:43.197586+0000","last_change":"2024-03-26T02:10:05.149442+0000","last_active":"2024-03-26T02:10:43.197586+0000","last_peered":"2024-03-26T02:10:43.197586+0000","last_clean":"2024-03-26T02:10:43.197586+0000","last_became_active":"2024-03-26T02:10:05.149091+0000","last_became_peered":"2024-03-26T02:10:05.149091+0000","last_unstale":"2024-03-26T02:10:43.197586+0000","last_undegraded":"2024-03-26T02:10:43.197586+0000","last_fullsized":"2024-03-26T02:10:43.197586+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:04:31.262421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347544+0000","last_change":"2024-03-26T02:10:07.165732+0000","last_active":"2024-03-26T02:10:31.347544+0000","last_peered":"2024-03-26T02:10:31.347544+0000","last_clean":"2024-03-26T02:10:31.347544+0000","last_became_active":"2024-03-26T02:10:07.165431+0000","last_became_peered":"2024-03-26T02:10:07.165431+0000","last_unstale":"2024-03-26T02:10:31.347544+0000","last_undegraded":"2024-03-26T02:10:31.347544+0000","last_fullsized":"2024-03-26T02:10:31.347544+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:11:41.518529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343161+0000","last_change":"2024-03-26T02:10:09.173428+0000","last_active":"2024-03-26T02:10:31.343161+0000","last_peered":"2024-03-26T02:10:31.343161+0000","last_clean":"2024-03-26T02:10:31.343161+0000","last_became_active":"2024-03-26T02:10:09.173043+0000","last_became_peered":"2024-03-26T02:10:09.173043+0000","last_unstale":"2024-03-26T02:10:31.343161+0000","last_undegraded":"2024-03-26T02:10:31.343161+0000","last_fullsized":"2024-03-26T02:10:31.343161+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:34.318077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345020+0000","last_change":"2024-03-26T02:10:03.146247+0000","last_active":"2024-03-26T02:10:31.345020+0000","last_peered":"2024-03-26T02:10:31.345020+0000","last_clean":"2024-03-26T02:10:31.345020+0000","last_became_active":"2024-03-26T02:10:03.145765+0000","last_became_peered":"2024-03-26T02:10:03.145765+0000","last_unstale":"2024-03-26T02:10:31.345020+0000","last_undegraded":"2024-03-26T02:10:31.345020+0000","last_fullsized":"2024-03-26T02:10:31.345020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:21:14.687378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'27","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346697+0000","last_change":"2024-03-26T02:10:05.153864+0000","last_active":"2024-03-26T02:10:31.346697+0000","last_peered":"2024-03-26T02:10:31.346697+0000","last_clean":"2024-03-26T02:10:31.346697+0000","last_became_active":"2024-03-26T02:10:05.153304+0000","last_became_peered":"2024-03-26T02:10:05.153304+0000","last_unstale":"2024-03-26T02:10:31.346697+0000","last_undegraded":"2024-03-26T02:10:31.346697+0000","last_fullsized":"2024-03-26T02:10:31.346697+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:01:15.263692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343601+0000","last_change":"2024-03-26T02:10:07.157515+0000","last_active":"2024-03-26T02:10:31.343601+0000","last_peered":"2024-03-26T02:10:31.343601+0000","last_clean":"2024-03-26T02:10:31.343601+0000","last_became_active":"2024-03-26T02:10:07.155926+0000","last_became_peered":"2024-03-26T02:10:07.155926+0000","last_unstale":"2024-03-26T02:10:31.343601+0000","last_undegraded":"2024-03-26T02:10:31.343601+0000","last_fullsized":"2024-03-26T02:10:31.343601+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:32:03.677085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345158+0000","last_change":"2024-03-26T02:10:09.176291+0000","last_active":"2024-03-26T02:10:31.345158+0000","last_peered":"2024-03-26T02:10:31.345158+0000","last_clean":"2024-03-26T02:10:31.345158+0000","last_became_active":"2024-03-26T02:10:09.175411+0000","last_became_peered":"2024-03-26T02:10:09.175411+0000","last_unstale":"2024-03-26T02:10:31.345158+0000","last_undegraded":"2024-03-26T02:10:31.345158+0000","last_fullsized":"2024-03-26T02:10:31.345158+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:09:28.241894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345318+0000","last_change":"2024-03-26T02:10:05.148986+0000","last_active":"2024-03-26T02:10:31.345318+0000","last_peered":"2024-03-26T02:10:31.345318+0000","last_clean":"2024-03-26T02:10:31.345318+0000","last_became_active":"2024-03-26T02:10:05.148293+0000","last_became_peered":"2024-03-26T02:10:05.148293+0000","last_unstale":"2024-03-26T02:10:31.345318+0000","last_undegraded":"2024-03-26T02:10:31.345318+0000","last_fullsized":"2024-03-26T02:10:31.345318+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:41:41.381981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342783+0000","last_change":"2024-03-26T02:10:03.146220+0000","last_active":"2024-03-26T02:10:31.342783+0000","last_peered":"2024-03-26T02:10:31.342783+0000","last_clean":"2024-03-26T02:10:31.342783+0000","last_became_active":"2024-03-26T02:10:03.145318+0000","last_became_peered":"2024-03-26T02:10:03.145318+0000","last_unstale":"2024-03-26T02:10:31.342783+0000","last_undegraded":"2024-03-26T02:10:31.342783+0000","last_fullsized":"2024-03-26T02:10:31.342783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:05:38.180067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345699+0000","last_change":"2024-03-26T02:10:07.158901+0000","last_active":"2024-03-26T02:10:31.345699+0000","last_peered":"2024-03-26T02:10:31.345699+0000","last_clean":"2024-03-26T02:10:31.345699+0000","last_became_active":"2024-03-26T02:10:07.158572+0000","last_became_peered":"2024-03-26T02:10:07.158572+0000","last_unstale":"2024-03-26T02:10:31.345699+0000","last_undegraded":"2024-03-26T02:10:31.345699+0000","last_fullsized":"2024-03-26T02:10:31.345699+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:20:02.303605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346969+0000","last_change":"2024-03-26T02:10:09.172226+0000","last_active":"2024-03-26T02:10:31.346969+0000","last_peered":"2024-03-26T02:10:31.346969+0000","last_clean":"2024-03-26T02:10:31.346969+0000","last_became_active":"2024-03-26T02:10:09.171923+0000","last_became_peered":"2024-03-26T02:10:09.171923+0000","last_unstale":"2024-03-26T02:10:31.346969+0000","last_undegraded":"2024-03-26T02:10:31.346969+0000","last_fullsized":"2024-03-26T02:10:31.346969+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:48.642310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342285+0000","last_change":"2024-03-26T02:10:05.148018+0000","last_active":"2024-03-26T02:10:31.342285+0000","last_peered":"2024-03-26T02:10:31.342285+0000","last_clean":"2024-03-26T02:10:31.342285+0000","last_became_active":"2024-03-26T02:10:05.147177+0000","last_became_peered":"2024-03-26T02:10:05.147177+0000","last_unstale":"2024-03-26T02:10:31.342285+0000","last_undegraded":"2024-03-26T02:10:31.342285+0000","last_fullsized":"2024-03-26T02:10:31.342285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:32:39.071397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345781+0000","last_change":"2024-03-26T02:10:03.147832+0000","last_active":"2024-03-26T02:10:31.345781+0000","last_peered":"2024-03-26T02:10:31.345781+0000","last_clean":"2024-03-26T02:10:31.345781+0000","last_became_active":"2024-03-26T02:10:03.146409+0000","last_became_peered":"2024-03-26T02:10:03.146409+0000","last_unstale":"2024-03-26T02:10:31.345781+0000","last_undegraded":"2024-03-26T02:10:31.345781+0000","last_fullsized":"2024-03-26T02:10:31.345781+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:48:13.557087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:33.457254+0000","last_change":"2024-03-26T02:09:52.036121+0000","last_active":"2024-03-26T02:10:33.457254+0000","last_peered":"2024-03-26T02:10:33.457254+0000","last_clean":"2024-03-26T02:10:33.457254+0000","last_became_active":"2024-03-26T02:09:52.035795+0000","last_became_peered":"2024-03-26T02:09:52.035795+0000","last_unstale":"2024-03-26T02:10:33.457254+0000","last_undegraded":"2024-03-26T02:10:33.457254+0000","last_fullsized":"2024-03-26T02:10:33.457254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:09:51.029144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:09:51.029144+0000","last_clean_scrub_stamp":"2024-03-26T02:09:51.029144+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:49:08.163765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346756+0000","last_change":"2024-03-26T02:10:07.157775+0000","last_active":"2024-03-26T02:10:31.346756+0000","last_peered":"2024-03-26T02:10:31.346756+0000","last_clean":"2024-03-26T02:10:31.346756+0000","last_became_active":"2024-03-26T02:10:07.157358+0000","last_became_peered":"2024-03-26T02:10:07.157358+0000","last_unstale":"2024-03-26T02:10:31.346756+0000","last_undegraded":"2024-03-26T02:10:31.346756+0000","last_fullsized":"2024-03-26T02:10:31.346756+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:08.392906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345051+0000","last_change":"2024-03-26T02:10:09.176510+0000","last_active":"2024-03-26T02:10:31.345051+0000","last_peered":"2024-03-26T02:10:31.345051+0000","last_clean":"2024-03-26T02:10:31.345051+0000","last_became_active":"2024-03-26T02:10:09.174022+0000","last_became_peered":"2024-03-26T02:10:09.174022+0000","last_unstale":"2024-03-26T02:10:31.345051+0000","last_undegraded":"2024-03-26T02:10:31.345051+0000","last_fullsized":"2024-03-26T02:10:31.345051+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:47:58.808062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344820+0000","last_change":"2024-03-26T02:10:05.151043+0000","last_active":"2024-03-26T02:10:31.344820+0000","last_peered":"2024-03-26T02:10:31.344820+0000","last_clean":"2024-03-26T02:10:31.344820+0000","last_became_active":"2024-03-26T02:10:05.150725+0000","last_became_peered":"2024-03-26T02:10:05.150725+0000","last_unstale":"2024-03-26T02:10:31.344820+0000","last_undegraded":"2024-03-26T02:10:31.344820+0000","last_fullsized":"2024-03-26T02:10:31.344820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:29:12.690541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343878+0000","last_change":"2024-03-26T02:10:03.148586+0000","last_active":"2024-03-26T02:10:31.343878+0000","last_peered":"2024-03-26T02:10:31.343878+0000","last_clean":"2024-03-26T02:10:31.343878+0000","last_became_active":"2024-03-26T02:10:03.148037+0000","last_became_peered":"2024-03-26T02:10:03.148037+0000","last_unstale":"2024-03-26T02:10:31.343878+0000","last_undegraded":"2024-03-26T02:10:31.343878+0000","last_fullsized":"2024-03-26T02:10:31.343878+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:48:20.731637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347581+0000","last_change":"2024-03-26T02:10:07.165653+0000","last_active":"2024-03-26T02:10:31.347581+0000","last_peered":"2024-03-26T02:10:31.347581+0000","last_clean":"2024-03-26T02:10:31.347581+0000","last_became_active":"2024-03-26T02:10:07.165130+0000","last_became_peered":"2024-03-26T02:10:07.165130+0000","last_unstale":"2024-03-26T02:10:31.347581+0000","last_undegraded":"2024-03-26T02:10:31.347581+0000","last_fullsized":"2024-03-26T02:10:31.347581+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:09:10.434457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345113+0000","last_change":"2024-03-26T02:10:09.176720+0000","last_active":"2024-03-26T02:10:31.345113+0000","last_peered":"2024-03-26T02:10:31.345113+0000","last_clean":"2024-03-26T02:10:31.345113+0000","last_became_active":"2024-03-26T02:10:09.176269+0000","last_became_peered":"2024-03-26T02:10:09.176269+0000","last_unstale":"2024-03-26T02:10:31.345113+0000","last_undegraded":"2024-03-26T02:10:31.345113+0000","last_fullsized":"2024-03-26T02:10:31.345113+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:59:41.438490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348362+0000","last_change":"2024-03-26T02:10:03.148495+0000","last_active":"2024-03-26T02:10:31.348362+0000","last_peered":"2024-03-26T02:10:31.348362+0000","last_clean":"2024-03-26T02:10:31.348362+0000","last_became_active":"2024-03-26T02:10:03.147232+0000","last_became_peered":"2024-03-26T02:10:03.147232+0000","last_unstale":"2024-03-26T02:10:31.348362+0000","last_undegraded":"2024-03-26T02:10:31.348362+0000","last_fullsized":"2024-03-26T02:10:31.348362+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:36:27.990383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'34","reported_seq":93,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:43.197372+0000","last_change":"2024-03-26T02:10:05.150785+0000","last_active":"2024-03-26T02:10:43.197372+0000","last_peered":"2024-03-26T02:10:43.197372+0000","last_clean":"2024-03-26T02:10:43.197372+0000","last_became_active":"2024-03-26T02:10:05.150409+0000","last_became_peered":"2024-03-26T02:10:05.150409+0000","last_unstale":"2024-03-26T02:10:43.197372+0000","last_undegraded":"2024-03-26T02:10:43.197372+0000","last_fullsized":"2024-03-26T02:10:43.197372+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:47:18.936093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:43.197445+0000","last_change":"2024-03-26T02:10:07.162869+0000","last_active":"2024-03-26T02:10:43.197445+0000","last_peered":"2024-03-26T02:10:43.197445+0000","last_clean":"2024-03-26T02:10:43.197445+0000","last_became_active":"2024-03-26T02:10:07.162526+0000","last_became_peered":"2024-03-26T02:10:07.162526+0000","last_unstale":"2024-03-26T02:10:43.197445+0000","last_undegraded":"2024-03-26T02:10:43.197445+0000","last_fullsized":"2024-03-26T02:10:43.197445+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:37:12.542294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344176+0000","last_change":"2024-03-26T02:10:09.178302+0000","last_active":"2024-03-26T02:10:31.344176+0000","last_peered":"2024-03-26T02:10:31.344176+0000","last_clean":"2024-03-26T02:10:31.344176+0000","last_became_active":"2024-03-26T02:10:09.177853+0000","last_became_peered":"2024-03-26T02:10:09.177853+0000","last_unstale":"2024-03-26T02:10:31.344176+0000","last_undegraded":"2024-03-26T02:10:31.344176+0000","last_fullsized":"2024-03-26T02:10:31.344176+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:28:22.954793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344789+0000","last_change":"2024-03-26T02:10:03.146268+0000","last_active":"2024-03-26T02:10:31.344789+0000","last_peered":"2024-03-26T02:10:31.344789+0000","last_clean":"2024-03-26T02:10:31.344789+0000","last_became_active":"2024-03-26T02:10:03.145238+0000","last_became_peered":"2024-03-26T02:10:03.145238+0000","last_unstale":"2024-03-26T02:10:31.344789+0000","last_undegraded":"2024-03-26T02:10:31.344789+0000","last_fullsized":"2024-03-26T02:10:31.344789+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:58:29.291845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345886+0000","last_change":"2024-03-26T02:10:05.152874+0000","last_active":"2024-03-26T02:10:31.345886+0000","last_peered":"2024-03-26T02:10:31.345886+0000","last_clean":"2024-03-26T02:10:31.345886+0000","last_became_active":"2024-03-26T02:10:05.152569+0000","last_became_peered":"2024-03-26T02:10:05.152569+0000","last_unstale":"2024-03-26T02:10:31.345886+0000","last_undegraded":"2024-03-26T02:10:31.345886+0000","last_fullsized":"2024-03-26T02:10:31.345886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:58.887192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:43.197298+0000","last_change":"2024-03-26T02:10:07.159627+0000","last_active":"2024-03-26T02:10:43.197298+0000","last_peered":"2024-03-26T02:10:43.197298+0000","last_clean":"2024-03-26T02:10:43.197298+0000","last_became_active":"2024-03-26T02:10:07.159169+0000","last_became_peered":"2024-03-26T02:10:07.159169+0000","last_unstale":"2024-03-26T02:10:43.197298+0000","last_undegraded":"2024-03-26T02:10:43.197298+0000","last_fullsized":"2024-03-26T02:10:43.197298+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:04:28.857380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346311+0000","last_change":"2024-03-26T02:10:09.178722+0000","last_active":"2024-03-26T02:10:31.346311+0000","last_peered":"2024-03-26T02:10:31.346311+0000","last_clean":"2024-03-26T02:10:31.346311+0000","last_became_active":"2024-03-26T02:10:09.178445+0000","last_became_peered":"2024-03-26T02:10:09.178445+0000","last_unstale":"2024-03-26T02:10:31.346311+0000","last_undegraded":"2024-03-26T02:10:31.346311+0000","last_fullsized":"2024-03-26T02:10:31.346311+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:21:42.960113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345457+0000","last_change":"2024-03-26T02:10:03.146592+0000","last_active":"2024-03-26T02:10:31.345457+0000","last_peered":"2024-03-26T02:10:31.345457+0000","last_clean":"2024-03-26T02:10:31.345457+0000","last_became_active":"2024-03-26T02:10:03.146186+0000","last_became_peered":"2024-03-26T02:10:03.146186+0000","last_unstale":"2024-03-26T02:10:31.345457+0000","last_undegraded":"2024-03-26T02:10:31.345457+0000","last_fullsized":"2024-03-26T02:10:31.345457+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:33:24.975689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344941+0000","last_change":"2024-03-26T02:10:05.152525+0000","last_active":"2024-03-26T02:10:31.344941+0000","last_peered":"2024-03-26T02:10:31.344941+0000","last_clean":"2024-03-26T02:10:31.344941+0000","last_became_active":"2024-03-26T02:10:05.151932+0000","last_became_peered":"2024-03-26T02:10:05.151932+0000","last_unstale":"2024-03-26T02:10:31.344941+0000","last_undegraded":"2024-03-26T02:10:31.344941+0000","last_fullsized":"2024-03-26T02:10:31.344941+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:44:37.133138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346429+0000","last_change":"2024-03-26T02:10:07.163466+0000","last_active":"2024-03-26T02:10:31.346429+0000","last_peered":"2024-03-26T02:10:31.346429+0000","last_clean":"2024-03-26T02:10:31.346429+0000","last_became_active":"2024-03-26T02:10:07.163179+0000","last_became_peered":"2024-03-26T02:10:07.163179+0000","last_unstale":"2024-03-26T02:10:31.346429+0000","last_undegraded":"2024-03-26T02:10:31.346429+0000","last_fullsized":"2024-03-26T02:10:31.346429+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:47:55.366104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344881+0000","last_change":"2024-03-26T02:10:09.176557+0000","last_active":"2024-03-26T02:10:31.344881+0000","last_peered":"2024-03-26T02:10:31.344881+0000","last_clean":"2024-03-26T02:10:31.344881+0000","last_became_active":"2024-03-26T02:10:09.172801+0000","last_became_peered":"2024-03-26T02:10:09.172801+0000","last_unstale":"2024-03-26T02:10:31.344881+0000","last_undegraded":"2024-03-26T02:10:31.344881+0000","last_fullsized":"2024-03-26T02:10:31.344881+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:57:55.215601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:43.197561+0000","last_change":"2024-03-26T02:10:07.155357+0000","last_active":"2024-03-26T02:10:43.197561+0000","last_peered":"2024-03-26T02:10:43.197561+0000","last_clean":"2024-03-26T02:10:43.197561+0000","last_became_active":"2024-03-26T02:10:07.155032+0000","last_became_peered":"2024-03-26T02:10:07.155032+0000","last_unstale":"2024-03-26T02:10:43.197561+0000","last_undegraded":"2024-03-26T02:10:43.197561+0000","last_fullsized":"2024-03-26T02:10:43.197561+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:20:25.573344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344000+0000","last_change":"2024-03-26T02:10:03.147704+0000","last_active":"2024-03-26T02:10:31.344000+0000","last_peered":"2024-03-26T02:10:31.344000+0000","last_clean":"2024-03-26T02:10:31.344000+0000","last_became_active":"2024-03-26T02:10:03.145871+0000","last_became_peered":"2024-03-26T02:10:03.145871+0000","last_unstale":"2024-03-26T02:10:31.344000+0000","last_undegraded":"2024-03-26T02:10:31.344000+0000","last_fullsized":"2024-03-26T02:10:31.344000+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:13:18.131271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346525+0000","last_change":"2024-03-26T02:10:05.148110+0000","last_active":"2024-03-26T02:10:31.346525+0000","last_peered":"2024-03-26T02:10:31.346525+0000","last_clean":"2024-03-26T02:10:31.346525+0000","last_became_active":"2024-03-26T02:10:05.147014+0000","last_became_peered":"2024-03-26T02:10:05.147014+0000","last_unstale":"2024-03-26T02:10:31.346525+0000","last_undegraded":"2024-03-26T02:10:31.346525+0000","last_fullsized":"2024-03-26T02:10:31.346525+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:52:09.112994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347711+0000","last_change":"2024-03-26T02:10:09.174800+0000","last_active":"2024-03-26T02:10:31.347711+0000","last_peered":"2024-03-26T02:10:31.347711+0000","last_clean":"2024-03-26T02:10:31.347711+0000","last_became_active":"2024-03-26T02:10:09.174506+0000","last_became_peered":"2024-03-26T02:10:09.174506+0000","last_unstale":"2024-03-26T02:10:31.347711+0000","last_undegraded":"2024-03-26T02:10:31.347711+0000","last_fullsized":"2024-03-26T02:10:31.347711+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:05:27.786474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345462+0000","last_change":"2024-03-26T02:10:07.157389+0000","last_active":"2024-03-26T02:10:31.345462+0000","last_peered":"2024-03-26T02:10:31.345462+0000","last_clean":"2024-03-26T02:10:31.345462+0000","last_became_active":"2024-03-26T02:10:07.156901+0000","last_became_peered":"2024-03-26T02:10:07.156901+0000","last_unstale":"2024-03-26T02:10:31.345462+0000","last_undegraded":"2024-03-26T02:10:31.345462+0000","last_fullsized":"2024-03-26T02:10:31.345462+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:00:53.748792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342615+0000","last_change":"2024-03-26T02:10:03.147000+0000","last_active":"2024-03-26T02:10:31.342615+0000","last_peered":"2024-03-26T02:10:31.342615+0000","last_clean":"2024-03-26T02:10:31.342615+0000","last_became_active":"2024-03-26T02:10:03.146464+0000","last_became_peered":"2024-03-26T02:10:03.146464+0000","last_unstale":"2024-03-26T02:10:31.342615+0000","last_undegraded":"2024-03-26T02:10:31.342615+0000","last_fullsized":"2024-03-26T02:10:31.342615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:53:09.795049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345966+0000","last_change":"2024-03-26T02:10:05.148745+0000","last_active":"2024-03-26T02:10:31.345966+0000","last_peered":"2024-03-26T02:10:31.345966+0000","last_clean":"2024-03-26T02:10:31.345966+0000","last_became_active":"2024-03-26T02:10:05.147575+0000","last_became_peered":"2024-03-26T02:10:05.147575+0000","last_unstale":"2024-03-26T02:10:31.345966+0000","last_undegraded":"2024-03-26T02:10:31.345966+0000","last_fullsized":"2024-03-26T02:10:31.345966+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:18:24.605202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347842+0000","last_change":"2024-03-26T02:10:09.171392+0000","last_active":"2024-03-26T02:10:31.347842+0000","last_peered":"2024-03-26T02:10:31.347842+0000","last_clean":"2024-03-26T02:10:31.347842+0000","last_became_active":"2024-03-26T02:10:09.171063+0000","last_became_peered":"2024-03-26T02:10:09.171063+0000","last_unstale":"2024-03-26T02:10:31.347842+0000","last_undegraded":"2024-03-26T02:10:31.347842+0000","last_fullsized":"2024-03-26T02:10:31.347842+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:01:01.822547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348113+0000","last_change":"2024-03-26T02:10:03.149001+0000","last_active":"2024-03-26T02:10:31.348113+0000","last_peered":"2024-03-26T02:10:31.348113+0000","last_clean":"2024-03-26T02:10:31.348113+0000","last_became_active":"2024-03-26T02:10:03.148093+0000","last_became_peered":"2024-03-26T02:10:03.148093+0000","last_unstale":"2024-03-26T02:10:31.348113+0000","last_undegraded":"2024-03-26T02:10:31.348113+0000","last_fullsized":"2024-03-26T02:10:31.348113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:51:55.430328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342889+0000","last_change":"2024-03-26T02:10:05.147551+0000","last_active":"2024-03-26T02:10:31.342889+0000","last_peered":"2024-03-26T02:10:31.342889+0000","last_clean":"2024-03-26T02:10:31.342889+0000","last_became_active":"2024-03-26T02:10:05.147194+0000","last_became_peered":"2024-03-26T02:10:05.147194+0000","last_unstale":"2024-03-26T02:10:31.342889+0000","last_undegraded":"2024-03-26T02:10:31.342889+0000","last_fullsized":"2024-03-26T02:10:31.342889+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:07:22.685556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345694+0000","last_change":"2024-03-26T02:10:07.154700+0000","last_active":"2024-03-26T02:10:31.345694+0000","last_peered":"2024-03-26T02:10:31.345694+0000","last_clean":"2024-03-26T02:10:31.345694+0000","last_became_active":"2024-03-26T02:10:07.154438+0000","last_became_peered":"2024-03-26T02:10:07.154438+0000","last_unstale":"2024-03-26T02:10:31.345694+0000","last_undegraded":"2024-03-26T02:10:31.345694+0000","last_fullsized":"2024-03-26T02:10:31.345694+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:23:17.868509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343243+0000","last_change":"2024-03-26T02:10:09.174105+0000","last_active":"2024-03-26T02:10:31.343243+0000","last_peered":"2024-03-26T02:10:31.343243+0000","last_clean":"2024-03-26T02:10:31.343243+0000","last_became_active":"2024-03-26T02:10:09.173440+0000","last_became_peered":"2024-03-26T02:10:09.173440+0000","last_unstale":"2024-03-26T02:10:31.343243+0000","last_undegraded":"2024-03-26T02:10:31.343243+0000","last_fullsized":"2024-03-26T02:10:31.343243+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:00:59.907859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:43.197570+0000","last_change":"2024-03-26T02:10:07.157698+0000","last_active":"2024-03-26T02:10:43.197570+0000","last_peered":"2024-03-26T02:10:43.197570+0000","last_clean":"2024-03-26T02:10:43.197570+0000","last_became_active":"2024-03-26T02:10:07.157349+0000","last_became_peered":"2024-03-26T02:10:07.157349+0000","last_unstale":"2024-03-26T02:10:43.197570+0000","last_undegraded":"2024-03-26T02:10:43.197570+0000","last_fullsized":"2024-03-26T02:10:43.197570+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:12:01.502132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346082+0000","last_change":"2024-03-26T02:10:03.148086+0000","last_active":"2024-03-26T02:10:31.346082+0000","last_peered":"2024-03-26T02:10:31.346082+0000","last_clean":"2024-03-26T02:10:31.346082+0000","last_became_active":"2024-03-26T02:10:03.147305+0000","last_became_peered":"2024-03-26T02:10:03.147305+0000","last_unstale":"2024-03-26T02:10:31.346082+0000","last_undegraded":"2024-03-26T02:10:31.346082+0000","last_fullsized":"2024-03-26T02:10:31.346082+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:55:57.277558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'26","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345334+0000","last_change":"2024-03-26T02:10:05.149623+0000","last_active":"2024-03-26T02:10:31.345334+0000","last_peered":"2024-03-26T02:10:31.345334+0000","last_clean":"2024-03-26T02:10:31.345334+0000","last_became_active":"2024-03-26T02:10:05.148175+0000","last_became_peered":"2024-03-26T02:10:05.148175+0000","last_unstale":"2024-03-26T02:10:31.345334+0000","last_undegraded":"2024-03-26T02:10:31.345334+0000","last_fullsized":"2024-03-26T02:10:31.345334+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:42:15.868815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345716+0000","last_change":"2024-03-26T02:10:09.174960+0000","last_active":"2024-03-26T02:10:31.345716+0000","last_peered":"2024-03-26T02:10:31.345716+0000","last_clean":"2024-03-26T02:10:31.345716+0000","last_became_active":"2024-03-26T02:10:09.174576+0000","last_became_peered":"2024-03-26T02:10:09.174576+0000","last_unstale":"2024-03-26T02:10:31.345716+0000","last_undegraded":"2024-03-26T02:10:31.345716+0000","last_fullsized":"2024-03-26T02:10:31.345716+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:53:29.187511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347568+0000","last_change":"2024-03-26T02:10:03.146340+0000","last_active":"2024-03-26T02:10:31.347568+0000","last_peered":"2024-03-26T02:10:31.347568+0000","last_clean":"2024-03-26T02:10:31.347568+0000","last_became_active":"2024-03-26T02:10:03.145926+0000","last_became_peered":"2024-03-26T02:10:03.145926+0000","last_unstale":"2024-03-26T02:10:31.347568+0000","last_undegraded":"2024-03-26T02:10:31.347568+0000","last_fullsized":"2024-03-26T02:10:31.347568+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:37:44.880105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347039+0000","last_change":"2024-03-26T02:10:05.149648+0000","last_active":"2024-03-26T02:10:31.347039+0000","last_peered":"2024-03-26T02:10:31.347039+0000","last_clean":"2024-03-26T02:10:31.347039+0000","last_became_active":"2024-03-26T02:10:05.149205+0000","last_became_peered":"2024-03-26T02:10:05.149205+0000","last_unstale":"2024-03-26T02:10:31.347039+0000","last_undegraded":"2024-03-26T02:10:31.347039+0000","last_fullsized":"2024-03-26T02:10:31.347039+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:54:37.358709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344811+0000","last_change":"2024-03-26T02:10:07.157966+0000","last_active":"2024-03-26T02:10:31.344811+0000","last_peered":"2024-03-26T02:10:31.344811+0000","last_clean":"2024-03-26T02:10:31.344811+0000","last_became_active":"2024-03-26T02:10:07.157233+0000","last_became_peered":"2024-03-26T02:10:07.157233+0000","last_unstale":"2024-03-26T02:10:31.344811+0000","last_undegraded":"2024-03-26T02:10:31.344811+0000","last_fullsized":"2024-03-26T02:10:31.344811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:13:23.456041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.341836+0000","last_change":"2024-03-26T02:10:09.172816+0000","last_active":"2024-03-26T02:10:31.341836+0000","last_peered":"2024-03-26T02:10:31.341836+0000","last_clean":"2024-03-26T02:10:31.341836+0000","last_became_active":"2024-03-26T02:10:09.172445+0000","last_became_peered":"2024-03-26T02:10:09.172445+0000","last_unstale":"2024-03-26T02:10:31.341836+0000","last_undegraded":"2024-03-26T02:10:31.341836+0000","last_fullsized":"2024-03-26T02:10:31.341836+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:02:22.260705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348186+0000","last_change":"2024-03-26T02:10:03.147972+0000","last_active":"2024-03-26T02:10:31.348186+0000","last_peered":"2024-03-26T02:10:31.348186+0000","last_clean":"2024-03-26T02:10:31.348186+0000","last_became_active":"2024-03-26T02:10:03.147224+0000","last_became_peered":"2024-03-26T02:10:03.147224+0000","last_unstale":"2024-03-26T02:10:31.348186+0000","last_undegraded":"2024-03-26T02:10:31.348186+0000","last_fullsized":"2024-03-26T02:10:31.348186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:42:27.433852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346767+0000","last_change":"2024-03-26T02:10:05.150266+0000","last_active":"2024-03-26T02:10:31.346767+0000","last_peered":"2024-03-26T02:10:31.346767+0000","last_clean":"2024-03-26T02:10:31.346767+0000","last_became_active":"2024-03-26T02:10:05.149898+0000","last_became_peered":"2024-03-26T02:10:05.149898+0000","last_unstale":"2024-03-26T02:10:31.346767+0000","last_undegraded":"2024-03-26T02:10:31.346767+0000","last_fullsized":"2024-03-26T02:10:31.346767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:36:39.044417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342574+0000","last_change":"2024-03-26T02:10:07.160503+0000","last_active":"2024-03-26T02:10:31.342574+0000","last_peered":"2024-03-26T02:10:31.342574+0000","last_clean":"2024-03-26T02:10:31.342574+0000","last_became_active":"2024-03-26T02:10:07.159992+0000","last_became_peered":"2024-03-26T02:10:07.159992+0000","last_unstale":"2024-03-26T02:10:31.342574+0000","last_undegraded":"2024-03-26T02:10:31.342574+0000","last_fullsized":"2024-03-26T02:10:31.342574+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:55:54.267028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343971+0000","last_change":"2024-03-26T02:10:09.176711+0000","last_active":"2024-03-26T02:10:31.343971+0000","last_peered":"2024-03-26T02:10:31.343971+0000","last_clean":"2024-03-26T02:10:31.343971+0000","last_became_active":"2024-03-26T02:10:09.176345+0000","last_became_peered":"2024-03-26T02:10:09.176345+0000","last_unstale":"2024-03-26T02:10:31.343971+0000","last_undegraded":"2024-03-26T02:10:31.343971+0000","last_fullsized":"2024-03-26T02:10:31.343971+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:07.347065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344615+0000","last_change":"2024-03-26T02:10:03.147091+0000","last_active":"2024-03-26T02:10:31.344615+0000","last_peered":"2024-03-26T02:10:31.344615+0000","last_clean":"2024-03-26T02:10:31.344615+0000","last_became_active":"2024-03-26T02:10:03.146748+0000","last_became_peered":"2024-03-26T02:10:03.146748+0000","last_unstale":"2024-03-26T02:10:31.344615+0000","last_undegraded":"2024-03-26T02:10:31.344615+0000","last_fullsized":"2024-03-26T02:10:31.344615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:36:40.299573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345542+0000","last_change":"2024-03-26T02:10:05.149723+0000","last_active":"2024-03-26T02:10:31.345542+0000","last_peered":"2024-03-26T02:10:31.345542+0000","last_clean":"2024-03-26T02:10:31.345542+0000","last_became_active":"2024-03-26T02:10:05.149266+0000","last_became_peered":"2024-03-26T02:10:05.149266+0000","last_unstale":"2024-03-26T02:10:31.345542+0000","last_undegraded":"2024-03-26T02:10:31.345542+0000","last_fullsized":"2024-03-26T02:10:31.345542+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:03:08.495256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346167+0000","last_change":"2024-03-26T02:10:07.164844+0000","last_active":"2024-03-26T02:10:31.346167+0000","last_peered":"2024-03-26T02:10:31.346167+0000","last_clean":"2024-03-26T02:10:31.346167+0000","last_became_active":"2024-03-26T02:10:07.163700+0000","last_became_peered":"2024-03-26T02:10:07.163700+0000","last_unstale":"2024-03-26T02:10:31.346167+0000","last_undegraded":"2024-03-26T02:10:31.346167+0000","last_fullsized":"2024-03-26T02:10:31.346167+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:11:09.664227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":589,"ondisk_log_size":589,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411343,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738394,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T02:10:55.845 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T02:10:56.425 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:10:56.516 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:56 smithi052 ceph-mon[25889]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:56.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:56 smithi165 ceph-mon[27627]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:56.754 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:56 smithi191 ceph-mon[27701]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:57.842 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:57 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ::ffff:172.21.15.165 - - [26/Mar/2024:02:10:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:10:57.843 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:57 smithi052 podman[49388]: 2024-03-26 02:10:57.750607397 +0000 UTC m=+0.135193094 volume create 46ac83ab827a8cc472c181c56565f1af3a704085e507ea76c362740f655f09c7 2024-03-26T02:10:57.843 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:57 smithi052 podman[49388]: 2024-03-26 02:10:57.842628781 +0000 UTC m=+0.227214479 container create 55187f8e13ad8edf039ee97bb0e222d1ab6afe2f899f1d1f1bd48822c5e4e86d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:10:58.690 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-mon[25889]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:58.690 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 podman[49388]: 2024-03-26 02:10:58.422840103 +0000 UTC m=+0.807425812 container init 55187f8e13ad8edf039ee97bb0e222d1ab6afe2f899f1d1f1bd48822c5e4e86d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:10:58.691 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 podman[49388]: 2024-03-26 02:10:58.429029998 +0000 UTC m=+0.813615695 container start 55187f8e13ad8edf039ee97bb0e222d1ab6afe2f899f1d1f1bd48822c5e4e86d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:10:58.691 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[49420]: ts=2024-03-26T02:10:58.442Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T02:10:58.691 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[49420]: ts=2024-03-26T02:10:58.442Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T02:10:58.691 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[49420]: ts=2024-03-26T02:10:58.443Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.52 port=9094 2024-03-26T02:10:58.691 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[49420]: ts=2024-03-26T02:10:58.444Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T02:10:58.691 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[49420]: ts=2024-03-26T02:10:58.468Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:10:58.691 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[49420]: ts=2024-03-26T02:10:58.469Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:10:58.691 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[49420]: ts=2024-03-26T02:10:58.471Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T02:10:58.691 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[49420]: ts=2024-03-26T02:10:58.471Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T02:10:58.691 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 bash[49388]: 55187f8e13ad8edf039ee97bb0e222d1ab6afe2f899f1d1f1bd48822c5e4e86d 2024-03-26T02:10:58.691 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:10:58 smithi052 systemd[1]: Started Ceph alertmanager.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:10:58.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:58 smithi165 ceph-mon[27627]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:58.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:58 smithi191 ceph-mon[27701]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:10:58.961 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:10:58] ENGINE Bus STOPPING 2024-03-26T02:10:58.961 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:10:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:10:58.961 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:10:58] ENGINE Bus STOPPED 2024-03-26T02:10:58.961 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:10:58] ENGINE Bus STARTING 2024-03-26T02:10:58.961 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:10:58] ENGINE Serving on http://:::9283 2024-03-26T02:10:58.961 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:10:58 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:10:58] ENGINE Bus STARTED 2024-03-26T02:10:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:59 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:59 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:59 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:59 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:59 smithi052 ceph-mon[25889]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T02:10:59.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:59 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:59 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:59 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:10:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:59 smithi052 ceph-mon[25889]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:10:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:59 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.712 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:10:59 smithi052 ceph-mon[25889]: Deploying daemon grafana.a on smithi165 2024-03-26T02:10:59.867 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:10:59.869 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-03-26T02:10:59.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:59 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:59 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:59 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:59 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:59 smithi165 ceph-mon[27627]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T02:10:59.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:59 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:59 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:59 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:10:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:59 smithi165 ceph-mon[27627]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:10:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:59 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:10:59.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:10:59 smithi165 ceph-mon[27627]: Deploying daemon grafana.a on smithi165 2024-03-26T02:11:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:59 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:59 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:59 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:59 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:59 smithi191 ceph-mon[27701]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T02:11:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:59 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:00.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:59 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:59 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:11:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:59 smithi191 ceph-mon[27701]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:11:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:59 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:00.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:10:59 smithi191 ceph-mon[27701]: Deploying daemon grafana.a on smithi165 2024-03-26T02:11:00.643 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:00 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[49420]: ts=2024-03-26T02:11:00.445Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001000416s 2024-03-26T02:11:00.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:00 smithi165 ceph-mon[27627]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:00.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:00 smithi165 ceph-mon[27627]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:11:00.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:00 smithi052 ceph-mon[25889]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:00.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:00 smithi052 ceph-mon[25889]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:11:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:00 smithi191 ceph-mon[27701]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:01.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:00 smithi191 ceph-mon[27701]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:11:01.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:01 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:01.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:01 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:01.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:01 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:01 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:01 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:02.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:01 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:02.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:01 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:02.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:01 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:02.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:01 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:02.457 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-03-26T02:10:59.357586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96612,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126556,"statfs":{"total":287972524032,"available":287873593344,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002790"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343751+0000","last_change":"2024-03-26T02:10:09.178848+0000","last_active":"2024-03-26T02:10:31.343751+0000","last_peered":"2024-03-26T02:10:31.343751+0000","last_clean":"2024-03-26T02:10:31.343751+0000","last_became_active":"2024-03-26T02:10:09.178491+0000","last_became_peered":"2024-03-26T02:10:09.178491+0000","last_unstale":"2024-03-26T02:10:31.343751+0000","last_undegraded":"2024-03-26T02:10:31.343751+0000","last_fullsized":"2024-03-26T02:10:31.343751+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:48:02.420816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345389+0000","last_change":"2024-03-26T02:10:07.162573+0000","last_active":"2024-03-26T02:10:31.345389+0000","last_peered":"2024-03-26T02:10:31.345389+0000","last_clean":"2024-03-26T02:10:31.345389+0000","last_became_active":"2024-03-26T02:10:07.161663+0000","last_became_peered":"2024-03-26T02:10:07.161663+0000","last_unstale":"2024-03-26T02:10:31.345389+0000","last_undegraded":"2024-03-26T02:10:31.345389+0000","last_fullsized":"2024-03-26T02:10:31.345389+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:58:58.822836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345612+0000","last_change":"2024-03-26T02:10:03.145978+0000","last_active":"2024-03-26T02:10:31.345612+0000","last_peered":"2024-03-26T02:10:31.345612+0000","last_clean":"2024-03-26T02:10:31.345612+0000","last_became_active":"2024-03-26T02:10:03.145444+0000","last_became_peered":"2024-03-26T02:10:03.145444+0000","last_unstale":"2024-03-26T02:10:31.345612+0000","last_undegraded":"2024-03-26T02:10:31.345612+0000","last_fullsized":"2024-03-26T02:10:31.345612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:53:01.171823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344587+0000","last_change":"2024-03-26T02:10:05.152356+0000","last_active":"2024-03-26T02:10:31.344587+0000","last_peered":"2024-03-26T02:10:31.344587+0000","last_clean":"2024-03-26T02:10:31.344587+0000","last_became_active":"2024-03-26T02:10:05.151925+0000","last_became_peered":"2024-03-26T02:10:05.151925+0000","last_unstale":"2024-03-26T02:10:31.344587+0000","last_undegraded":"2024-03-26T02:10:31.344587+0000","last_fullsized":"2024-03-26T02:10:31.344587+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:45:46.319407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342287+0000","last_change":"2024-03-26T02:10:09.172611+0000","last_active":"2024-03-26T02:10:31.342287+0000","last_peered":"2024-03-26T02:10:31.342287+0000","last_clean":"2024-03-26T02:10:31.342287+0000","last_became_active":"2024-03-26T02:10:09.172327+0000","last_became_peered":"2024-03-26T02:10:09.172327+0000","last_unstale":"2024-03-26T02:10:31.342287+0000","last_undegraded":"2024-03-26T02:10:31.342287+0000","last_fullsized":"2024-03-26T02:10:31.342287+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:26:24.560661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344225+0000","last_change":"2024-03-26T02:10:07.158999+0000","last_active":"2024-03-26T02:10:31.344225+0000","last_peered":"2024-03-26T02:10:31.344225+0000","last_clean":"2024-03-26T02:10:31.344225+0000","last_became_active":"2024-03-26T02:10:07.157751+0000","last_became_peered":"2024-03-26T02:10:07.157751+0000","last_unstale":"2024-03-26T02:10:31.344225+0000","last_undegraded":"2024-03-26T02:10:31.344225+0000","last_fullsized":"2024-03-26T02:10:31.344225+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:48:48.875583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345072+0000","last_change":"2024-03-26T02:10:03.146351+0000","last_active":"2024-03-26T02:10:31.345072+0000","last_peered":"2024-03-26T02:10:31.345072+0000","last_clean":"2024-03-26T02:10:31.345072+0000","last_became_active":"2024-03-26T02:10:03.145242+0000","last_became_peered":"2024-03-26T02:10:03.145242+0000","last_unstale":"2024-03-26T02:10:31.345072+0000","last_undegraded":"2024-03-26T02:10:31.345072+0000","last_fullsized":"2024-03-26T02:10:31.345072+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:32:18.434819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345555+0000","last_change":"2024-03-26T02:10:05.151212+0000","last_active":"2024-03-26T02:10:31.345555+0000","last_peered":"2024-03-26T02:10:31.345555+0000","last_clean":"2024-03-26T02:10:31.345555+0000","last_became_active":"2024-03-26T02:10:05.150844+0000","last_became_peered":"2024-03-26T02:10:05.150844+0000","last_unstale":"2024-03-26T02:10:31.345555+0000","last_undegraded":"2024-03-26T02:10:31.345555+0000","last_fullsized":"2024-03-26T02:10:31.345555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:47:21.893622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346808+0000","last_change":"2024-03-26T02:10:09.171246+0000","last_active":"2024-03-26T02:10:31.346808+0000","last_peered":"2024-03-26T02:10:31.346808+0000","last_clean":"2024-03-26T02:10:31.346808+0000","last_became_active":"2024-03-26T02:10:09.170781+0000","last_became_peered":"2024-03-26T02:10:09.170781+0000","last_unstale":"2024-03-26T02:10:31.346808+0000","last_undegraded":"2024-03-26T02:10:31.346808+0000","last_fullsized":"2024-03-26T02:10:31.346808+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:41:00.578858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344327+0000","last_change":"2024-03-26T02:10:07.160672+0000","last_active":"2024-03-26T02:10:31.344327+0000","last_peered":"2024-03-26T02:10:31.344327+0000","last_clean":"2024-03-26T02:10:31.344327+0000","last_became_active":"2024-03-26T02:10:07.159880+0000","last_became_peered":"2024-03-26T02:10:07.159880+0000","last_unstale":"2024-03-26T02:10:31.344327+0000","last_undegraded":"2024-03-26T02:10:31.344327+0000","last_fullsized":"2024-03-26T02:10:31.344327+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:10:46.621557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348597+0000","last_change":"2024-03-26T02:10:03.149069+0000","last_active":"2024-03-26T02:10:31.348597+0000","last_peered":"2024-03-26T02:10:31.348597+0000","last_clean":"2024-03-26T02:10:31.348597+0000","last_became_active":"2024-03-26T02:10:03.147674+0000","last_became_peered":"2024-03-26T02:10:03.147674+0000","last_unstale":"2024-03-26T02:10:31.348597+0000","last_undegraded":"2024-03-26T02:10:31.348597+0000","last_fullsized":"2024-03-26T02:10:31.348597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:42:21.413893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346930+0000","last_change":"2024-03-26T02:10:05.151500+0000","last_active":"2024-03-26T02:10:31.346930+0000","last_peered":"2024-03-26T02:10:31.346930+0000","last_clean":"2024-03-26T02:10:31.346930+0000","last_became_active":"2024-03-26T02:10:05.151163+0000","last_became_peered":"2024-03-26T02:10:05.151163+0000","last_unstale":"2024-03-26T02:10:31.346930+0000","last_undegraded":"2024-03-26T02:10:31.346930+0000","last_fullsized":"2024-03-26T02:10:31.346930+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:22.146610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344043+0000","last_change":"2024-03-26T02:10:09.180164+0000","last_active":"2024-03-26T02:10:31.344043+0000","last_peered":"2024-03-26T02:10:31.344043+0000","last_clean":"2024-03-26T02:10:31.344043+0000","last_became_active":"2024-03-26T02:10:09.179725+0000","last_became_peered":"2024-03-26T02:10:09.179725+0000","last_unstale":"2024-03-26T02:10:31.344043+0000","last_undegraded":"2024-03-26T02:10:31.344043+0000","last_fullsized":"2024-03-26T02:10:31.344043+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:10:42.248845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344669+0000","last_change":"2024-03-26T02:10:07.158187+0000","last_active":"2024-03-26T02:10:31.344669+0000","last_peered":"2024-03-26T02:10:31.344669+0000","last_clean":"2024-03-26T02:10:31.344669+0000","last_became_active":"2024-03-26T02:10:07.157653+0000","last_became_peered":"2024-03-26T02:10:07.157653+0000","last_unstale":"2024-03-26T02:10:31.344669+0000","last_undegraded":"2024-03-26T02:10:31.344669+0000","last_fullsized":"2024-03-26T02:10:31.344669+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:45:43.729359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347361+0000","last_change":"2024-03-26T02:10:03.145736+0000","last_active":"2024-03-26T02:10:31.347361+0000","last_peered":"2024-03-26T02:10:31.347361+0000","last_clean":"2024-03-26T02:10:31.347361+0000","last_became_active":"2024-03-26T02:10:03.145404+0000","last_became_peered":"2024-03-26T02:10:03.145404+0000","last_unstale":"2024-03-26T02:10:31.347361+0000","last_undegraded":"2024-03-26T02:10:31.347361+0000","last_fullsized":"2024-03-26T02:10:31.347361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:04:18.823386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'18","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348466+0000","last_change":"2024-03-26T02:10:05.149031+0000","last_active":"2024-03-26T02:10:31.348466+0000","last_peered":"2024-03-26T02:10:31.348466+0000","last_clean":"2024-03-26T02:10:31.348466+0000","last_became_active":"2024-03-26T02:10:05.148447+0000","last_became_peered":"2024-03-26T02:10:05.148447+0000","last_unstale":"2024-03-26T02:10:31.348466+0000","last_undegraded":"2024-03-26T02:10:31.348466+0000","last_fullsized":"2024-03-26T02:10:31.348466+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:37:38.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345581+0000","last_change":"2024-03-26T02:10:09.180017+0000","last_active":"2024-03-26T02:10:31.345581+0000","last_peered":"2024-03-26T02:10:31.345581+0000","last_clean":"2024-03-26T02:10:31.345581+0000","last_became_active":"2024-03-26T02:10:09.179644+0000","last_became_peered":"2024-03-26T02:10:09.179644+0000","last_unstale":"2024-03-26T02:10:31.345581+0000","last_undegraded":"2024-03-26T02:10:31.345581+0000","last_fullsized":"2024-03-26T02:10:31.345581+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:20:16.575058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348078+0000","last_change":"2024-03-26T02:10:07.162457+0000","last_active":"2024-03-26T02:10:31.348078+0000","last_peered":"2024-03-26T02:10:31.348078+0000","last_clean":"2024-03-26T02:10:31.348078+0000","last_became_active":"2024-03-26T02:10:07.161593+0000","last_became_peered":"2024-03-26T02:10:07.161593+0000","last_unstale":"2024-03-26T02:10:31.348078+0000","last_undegraded":"2024-03-26T02:10:31.348078+0000","last_fullsized":"2024-03-26T02:10:31.348078+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:08:49.788507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346178+0000","last_change":"2024-03-26T02:10:03.147724+0000","last_active":"2024-03-26T02:10:31.346178+0000","last_peered":"2024-03-26T02:10:31.346178+0000","last_clean":"2024-03-26T02:10:31.346178+0000","last_became_active":"2024-03-26T02:10:03.145771+0000","last_became_peered":"2024-03-26T02:10:03.145771+0000","last_unstale":"2024-03-26T02:10:31.346178+0000","last_undegraded":"2024-03-26T02:10:31.346178+0000","last_fullsized":"2024-03-26T02:10:31.346178+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:16:33.102380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'22","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348229+0000","last_change":"2024-03-26T02:10:05.147593+0000","last_active":"2024-03-26T02:10:31.348229+0000","last_peered":"2024-03-26T02:10:31.348229+0000","last_clean":"2024-03-26T02:10:31.348229+0000","last_became_active":"2024-03-26T02:10:05.147292+0000","last_became_peered":"2024-03-26T02:10:05.147292+0000","last_unstale":"2024-03-26T02:10:31.348229+0000","last_undegraded":"2024-03-26T02:10:31.348229+0000","last_fullsized":"2024-03-26T02:10:31.348229+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:22:28.271533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343780+0000","last_change":"2024-03-26T02:10:09.174779+0000","last_active":"2024-03-26T02:10:31.343780+0000","last_peered":"2024-03-26T02:10:31.343780+0000","last_clean":"2024-03-26T02:10:31.343780+0000","last_became_active":"2024-03-26T02:10:09.174433+0000","last_became_peered":"2024-03-26T02:10:09.174433+0000","last_unstale":"2024-03-26T02:10:31.343780+0000","last_undegraded":"2024-03-26T02:10:31.343780+0000","last_fullsized":"2024-03-26T02:10:31.343780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:15:05.523966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347174+0000","last_change":"2024-03-26T02:10:07.162758+0000","last_active":"2024-03-26T02:10:31.347174+0000","last_peered":"2024-03-26T02:10:31.347174+0000","last_clean":"2024-03-26T02:10:31.347174+0000","last_became_active":"2024-03-26T02:10:07.161766+0000","last_became_peered":"2024-03-26T02:10:07.161766+0000","last_unstale":"2024-03-26T02:10:31.347174+0000","last_undegraded":"2024-03-26T02:10:31.347174+0000","last_fullsized":"2024-03-26T02:10:31.347174+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:14:59.629464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345198+0000","last_change":"2024-03-26T02:10:03.146476+0000","last_active":"2024-03-26T02:10:31.345198+0000","last_peered":"2024-03-26T02:10:31.345198+0000","last_clean":"2024-03-26T02:10:31.345198+0000","last_became_active":"2024-03-26T02:10:03.145883+0000","last_became_peered":"2024-03-26T02:10:03.145883+0000","last_unstale":"2024-03-26T02:10:31.345198+0000","last_undegraded":"2024-03-26T02:10:31.345198+0000","last_fullsized":"2024-03-26T02:10:31.345198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:28:25.046266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346031+0000","last_change":"2024-03-26T02:10:05.148750+0000","last_active":"2024-03-26T02:10:31.346031+0000","last_peered":"2024-03-26T02:10:31.346031+0000","last_clean":"2024-03-26T02:10:31.346031+0000","last_became_active":"2024-03-26T02:10:05.148267+0000","last_became_peered":"2024-03-26T02:10:05.148267+0000","last_unstale":"2024-03-26T02:10:31.346031+0000","last_undegraded":"2024-03-26T02:10:31.346031+0000","last_fullsized":"2024-03-26T02:10:31.346031+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:57:14.794777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344379+0000","last_change":"2024-03-26T02:10:09.176305+0000","last_active":"2024-03-26T02:10:31.344379+0000","last_peered":"2024-03-26T02:10:31.344379+0000","last_clean":"2024-03-26T02:10:31.344379+0000","last_became_active":"2024-03-26T02:10:09.175277+0000","last_became_peered":"2024-03-26T02:10:09.175277+0000","last_unstale":"2024-03-26T02:10:31.344379+0000","last_undegraded":"2024-03-26T02:10:31.344379+0000","last_fullsized":"2024-03-26T02:10:31.344379+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:16:52.365975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:48.198587+0000","last_change":"2024-03-26T02:10:07.165428+0000","last_active":"2024-03-26T02:10:48.198587+0000","last_peered":"2024-03-26T02:10:48.198587+0000","last_clean":"2024-03-26T02:10:48.198587+0000","last_became_active":"2024-03-26T02:10:07.164967+0000","last_became_peered":"2024-03-26T02:10:07.164967+0000","last_unstale":"2024-03-26T02:10:48.198587+0000","last_undegraded":"2024-03-26T02:10:48.198587+0000","last_fullsized":"2024-03-26T02:10:48.198587+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:17:30.903932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344719+0000","last_change":"2024-03-26T02:10:03.146758+0000","last_active":"2024-03-26T02:10:31.344719+0000","last_peered":"2024-03-26T02:10:31.344719+0000","last_clean":"2024-03-26T02:10:31.344719+0000","last_became_active":"2024-03-26T02:10:03.145491+0000","last_became_peered":"2024-03-26T02:10:03.145491+0000","last_unstale":"2024-03-26T02:10:31.344719+0000","last_undegraded":"2024-03-26T02:10:31.344719+0000","last_fullsized":"2024-03-26T02:10:31.344719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:04:08.395675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342019+0000","last_change":"2024-03-26T02:10:05.151580+0000","last_active":"2024-03-26T02:10:31.342019+0000","last_peered":"2024-03-26T02:10:31.342019+0000","last_clean":"2024-03-26T02:10:31.342019+0000","last_became_active":"2024-03-26T02:10:05.151179+0000","last_became_peered":"2024-03-26T02:10:05.151179+0000","last_unstale":"2024-03-26T02:10:31.342019+0000","last_undegraded":"2024-03-26T02:10:31.342019+0000","last_fullsized":"2024-03-26T02:10:31.342019+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:11:16.297115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344660+0000","last_change":"2024-03-26T02:10:09.178215+0000","last_active":"2024-03-26T02:10:31.344660+0000","last_peered":"2024-03-26T02:10:31.344660+0000","last_clean":"2024-03-26T02:10:31.344660+0000","last_became_active":"2024-03-26T02:10:09.177638+0000","last_became_peered":"2024-03-26T02:10:09.177638+0000","last_unstale":"2024-03-26T02:10:31.344660+0000","last_undegraded":"2024-03-26T02:10:31.344660+0000","last_fullsized":"2024-03-26T02:10:31.344660+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:47:18.325957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344444+0000","last_change":"2024-03-26T02:10:07.156833+0000","last_active":"2024-03-26T02:10:31.344444+0000","last_peered":"2024-03-26T02:10:31.344444+0000","last_clean":"2024-03-26T02:10:31.344444+0000","last_became_active":"2024-03-26T02:10:07.156018+0000","last_became_peered":"2024-03-26T02:10:07.156018+0000","last_unstale":"2024-03-26T02:10:31.344444+0000","last_undegraded":"2024-03-26T02:10:31.344444+0000","last_fullsized":"2024-03-26T02:10:31.344444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:29:16.614231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345530+0000","last_change":"2024-03-26T02:10:03.149104+0000","last_active":"2024-03-26T02:10:31.345530+0000","last_peered":"2024-03-26T02:10:31.345530+0000","last_clean":"2024-03-26T02:10:31.345530+0000","last_became_active":"2024-03-26T02:10:03.147860+0000","last_became_peered":"2024-03-26T02:10:03.147860+0000","last_unstale":"2024-03-26T02:10:31.345530+0000","last_undegraded":"2024-03-26T02:10:31.345530+0000","last_fullsized":"2024-03-26T02:10:31.345530+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:13:48.500636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'39","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342020+0000","last_change":"2024-03-26T02:10:05.150313+0000","last_active":"2024-03-26T02:10:31.342020+0000","last_peered":"2024-03-26T02:10:31.342020+0000","last_clean":"2024-03-26T02:10:31.342020+0000","last_became_active":"2024-03-26T02:10:05.149889+0000","last_became_peered":"2024-03-26T02:10:05.149889+0000","last_unstale":"2024-03-26T02:10:31.342020+0000","last_undegraded":"2024-03-26T02:10:31.342020+0000","last_fullsized":"2024-03-26T02:10:31.342020+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:49:15.417727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344510+0000","last_change":"2024-03-26T02:10:09.180826+0000","last_active":"2024-03-26T02:10:31.344510+0000","last_peered":"2024-03-26T02:10:31.344510+0000","last_clean":"2024-03-26T02:10:31.344510+0000","last_became_active":"2024-03-26T02:10:09.180484+0000","last_became_peered":"2024-03-26T02:10:09.180484+0000","last_unstale":"2024-03-26T02:10:31.344510+0000","last_undegraded":"2024-03-26T02:10:31.344510+0000","last_fullsized":"2024-03-26T02:10:31.344510+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:08:35.589827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.341894+0000","last_change":"2024-03-26T02:10:07.155543+0000","last_active":"2024-03-26T02:10:31.341894+0000","last_peered":"2024-03-26T02:10:31.341894+0000","last_clean":"2024-03-26T02:10:31.341894+0000","last_became_active":"2024-03-26T02:10:07.155227+0000","last_became_peered":"2024-03-26T02:10:07.155227+0000","last_unstale":"2024-03-26T02:10:31.341894+0000","last_undegraded":"2024-03-26T02:10:31.341894+0000","last_fullsized":"2024-03-26T02:10:31.341894+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:11:33.780128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345918+0000","last_change":"2024-03-26T02:10:03.148045+0000","last_active":"2024-03-26T02:10:31.345918+0000","last_peered":"2024-03-26T02:10:31.345918+0000","last_clean":"2024-03-26T02:10:31.345918+0000","last_became_active":"2024-03-26T02:10:03.147108+0000","last_became_peered":"2024-03-26T02:10:03.147108+0000","last_unstale":"2024-03-26T02:10:31.345918+0000","last_undegraded":"2024-03-26T02:10:31.345918+0000","last_fullsized":"2024-03-26T02:10:31.345918+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:45:15.645475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345283+0000","last_change":"2024-03-26T02:10:05.149174+0000","last_active":"2024-03-26T02:10:31.345283+0000","last_peered":"2024-03-26T02:10:31.345283+0000","last_clean":"2024-03-26T02:10:31.345283+0000","last_became_active":"2024-03-26T02:10:05.148232+0000","last_became_peered":"2024-03-26T02:10:05.148232+0000","last_unstale":"2024-03-26T02:10:31.345283+0000","last_undegraded":"2024-03-26T02:10:31.345283+0000","last_fullsized":"2024-03-26T02:10:31.345283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:06:07.747446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345632+0000","last_change":"2024-03-26T02:10:09.176384+0000","last_active":"2024-03-26T02:10:31.345632+0000","last_peered":"2024-03-26T02:10:31.345632+0000","last_clean":"2024-03-26T02:10:31.345632+0000","last_became_active":"2024-03-26T02:10:09.175965+0000","last_became_peered":"2024-03-26T02:10:09.175965+0000","last_unstale":"2024-03-26T02:10:31.345632+0000","last_undegraded":"2024-03-26T02:10:31.345632+0000","last_fullsized":"2024-03-26T02:10:31.345632+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:50:24.176240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342319+0000","last_change":"2024-03-26T02:10:07.157393+0000","last_active":"2024-03-26T02:10:31.342319+0000","last_peered":"2024-03-26T02:10:31.342319+0000","last_clean":"2024-03-26T02:10:31.342319+0000","last_became_active":"2024-03-26T02:10:07.156932+0000","last_became_peered":"2024-03-26T02:10:07.156932+0000","last_unstale":"2024-03-26T02:10:31.342319+0000","last_undegraded":"2024-03-26T02:10:31.342319+0000","last_fullsized":"2024-03-26T02:10:31.342319+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:44:28.689508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345023+0000","last_change":"2024-03-26T02:10:03.150694+0000","last_active":"2024-03-26T02:10:31.345023+0000","last_peered":"2024-03-26T02:10:31.345023+0000","last_clean":"2024-03-26T02:10:31.345023+0000","last_became_active":"2024-03-26T02:10:03.150121+0000","last_became_peered":"2024-03-26T02:10:03.150121+0000","last_unstale":"2024-03-26T02:10:31.345023+0000","last_undegraded":"2024-03-26T02:10:31.345023+0000","last_fullsized":"2024-03-26T02:10:31.345023+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:11:17.951007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345765+0000","last_change":"2024-03-26T02:10:05.151468+0000","last_active":"2024-03-26T02:10:31.345765+0000","last_peered":"2024-03-26T02:10:31.345765+0000","last_clean":"2024-03-26T02:10:31.345765+0000","last_became_active":"2024-03-26T02:10:05.151035+0000","last_became_peered":"2024-03-26T02:10:05.151035+0000","last_unstale":"2024-03-26T02:10:31.345765+0000","last_undegraded":"2024-03-26T02:10:31.345765+0000","last_fullsized":"2024-03-26T02:10:31.345765+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:38:43.396252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346963+0000","last_change":"2024-03-26T02:10:09.171573+0000","last_active":"2024-03-26T02:10:31.346963+0000","last_peered":"2024-03-26T02:10:31.346963+0000","last_clean":"2024-03-26T02:10:31.346963+0000","last_became_active":"2024-03-26T02:10:09.171283+0000","last_became_peered":"2024-03-26T02:10:09.171283+0000","last_unstale":"2024-03-26T02:10:31.346963+0000","last_undegraded":"2024-03-26T02:10:31.346963+0000","last_fullsized":"2024-03-26T02:10:31.346963+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:50:36.130678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344505+0000","last_change":"2024-03-26T02:10:07.159762+0000","last_active":"2024-03-26T02:10:31.344505+0000","last_peered":"2024-03-26T02:10:31.344505+0000","last_clean":"2024-03-26T02:10:31.344505+0000","last_became_active":"2024-03-26T02:10:07.159344+0000","last_became_peered":"2024-03-26T02:10:07.159344+0000","last_unstale":"2024-03-26T02:10:31.344505+0000","last_undegraded":"2024-03-26T02:10:31.344505+0000","last_fullsized":"2024-03-26T02:10:31.344505+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:14:00.267421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348457+0000","last_change":"2024-03-26T02:10:03.147577+0000","last_active":"2024-03-26T02:10:31.348457+0000","last_peered":"2024-03-26T02:10:31.348457+0000","last_clean":"2024-03-26T02:10:31.348457+0000","last_became_active":"2024-03-26T02:10:03.146249+0000","last_became_peered":"2024-03-26T02:10:03.146249+0000","last_unstale":"2024-03-26T02:10:31.348457+0000","last_undegraded":"2024-03-26T02:10:31.348457+0000","last_fullsized":"2024-03-26T02:10:31.348457+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:15:18.236497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346893+0000","last_change":"2024-03-26T02:10:05.149586+0000","last_active":"2024-03-26T02:10:31.346893+0000","last_peered":"2024-03-26T02:10:31.346893+0000","last_clean":"2024-03-26T02:10:31.346893+0000","last_became_active":"2024-03-26T02:10:05.148951+0000","last_became_peered":"2024-03-26T02:10:05.148951+0000","last_unstale":"2024-03-26T02:10:31.346893+0000","last_undegraded":"2024-03-26T02:10:31.346893+0000","last_fullsized":"2024-03-26T02:10:31.346893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:25:42.506433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345597+0000","last_change":"2024-03-26T02:10:09.178976+0000","last_active":"2024-03-26T02:10:31.345597+0000","last_peered":"2024-03-26T02:10:31.345597+0000","last_clean":"2024-03-26T02:10:31.345597+0000","last_became_active":"2024-03-26T02:10:09.178618+0000","last_became_peered":"2024-03-26T02:10:09.178618+0000","last_unstale":"2024-03-26T02:10:31.345597+0000","last_undegraded":"2024-03-26T02:10:31.345597+0000","last_fullsized":"2024-03-26T02:10:31.345597+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:52:13.605859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342190+0000","last_change":"2024-03-26T02:10:07.156494+0000","last_active":"2024-03-26T02:10:31.342190+0000","last_peered":"2024-03-26T02:10:31.342190+0000","last_clean":"2024-03-26T02:10:31.342190+0000","last_became_active":"2024-03-26T02:10:07.156153+0000","last_became_peered":"2024-03-26T02:10:07.156153+0000","last_unstale":"2024-03-26T02:10:31.342190+0000","last_undegraded":"2024-03-26T02:10:31.342190+0000","last_fullsized":"2024-03-26T02:10:31.342190+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:21:42.466813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346099+0000","last_change":"2024-03-26T02:10:03.150380+0000","last_active":"2024-03-26T02:10:31.346099+0000","last_peered":"2024-03-26T02:10:31.346099+0000","last_clean":"2024-03-26T02:10:31.346099+0000","last_became_active":"2024-03-26T02:10:03.150047+0000","last_became_peered":"2024-03-26T02:10:03.150047+0000","last_unstale":"2024-03-26T02:10:31.346099+0000","last_undegraded":"2024-03-26T02:10:31.346099+0000","last_fullsized":"2024-03-26T02:10:31.346099+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:47:40.410519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348279+0000","last_change":"2024-03-26T02:10:05.148761+0000","last_active":"2024-03-26T02:10:31.348279+0000","last_peered":"2024-03-26T02:10:31.348279+0000","last_clean":"2024-03-26T02:10:31.348279+0000","last_became_active":"2024-03-26T02:10:05.148503+0000","last_became_peered":"2024-03-26T02:10:05.148503+0000","last_unstale":"2024-03-26T02:10:31.348279+0000","last_undegraded":"2024-03-26T02:10:31.348279+0000","last_fullsized":"2024-03-26T02:10:31.348279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:28:31.258420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343511+0000","last_change":"2024-03-26T02:10:09.176185+0000","last_active":"2024-03-26T02:10:31.343511+0000","last_peered":"2024-03-26T02:10:31.343511+0000","last_clean":"2024-03-26T02:10:31.343511+0000","last_became_active":"2024-03-26T02:10:09.174980+0000","last_became_peered":"2024-03-26T02:10:09.174980+0000","last_unstale":"2024-03-26T02:10:31.343511+0000","last_undegraded":"2024-03-26T02:10:31.343511+0000","last_fullsized":"2024-03-26T02:10:31.343511+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:42:54.816228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:48.198621+0000","last_change":"2024-03-26T02:10:07.163792+0000","last_active":"2024-03-26T02:10:48.198621+0000","last_peered":"2024-03-26T02:10:48.198621+0000","last_clean":"2024-03-26T02:10:48.198621+0000","last_became_active":"2024-03-26T02:10:07.163431+0000","last_became_peered":"2024-03-26T02:10:07.163431+0000","last_unstale":"2024-03-26T02:10:48.198621+0000","last_undegraded":"2024-03-26T02:10:48.198621+0000","last_fullsized":"2024-03-26T02:10:48.198621+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:53:46.495755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345936+0000","last_change":"2024-03-26T02:10:03.148408+0000","last_active":"2024-03-26T02:10:31.345936+0000","last_peered":"2024-03-26T02:10:31.345936+0000","last_clean":"2024-03-26T02:10:31.345936+0000","last_became_active":"2024-03-26T02:10:03.147567+0000","last_became_peered":"2024-03-26T02:10:03.147567+0000","last_unstale":"2024-03-26T02:10:31.345936+0000","last_undegraded":"2024-03-26T02:10:31.345936+0000","last_fullsized":"2024-03-26T02:10:31.345936+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:37:57.113288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342450+0000","last_change":"2024-03-26T02:10:05.150995+0000","last_active":"2024-03-26T02:10:31.342450+0000","last_peered":"2024-03-26T02:10:31.342450+0000","last_clean":"2024-03-26T02:10:31.342450+0000","last_became_active":"2024-03-26T02:10:05.150551+0000","last_became_peered":"2024-03-26T02:10:05.150551+0000","last_unstale":"2024-03-26T02:10:31.342450+0000","last_undegraded":"2024-03-26T02:10:31.342450+0000","last_fullsized":"2024-03-26T02:10:31.342450+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:10:22.839667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345753+0000","last_change":"2024-03-26T02:10:09.180546+0000","last_active":"2024-03-26T02:10:31.345753+0000","last_peered":"2024-03-26T02:10:31.345753+0000","last_clean":"2024-03-26T02:10:31.345753+0000","last_became_active":"2024-03-26T02:10:09.180012+0000","last_became_peered":"2024-03-26T02:10:09.180012+0000","last_unstale":"2024-03-26T02:10:31.345753+0000","last_undegraded":"2024-03-26T02:10:31.345753+0000","last_fullsized":"2024-03-26T02:10:31.345753+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:54:37.899012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:48.198601+0000","last_change":"2024-03-26T02:10:07.162776+0000","last_active":"2024-03-26T02:10:48.198601+0000","last_peered":"2024-03-26T02:10:48.198601+0000","last_clean":"2024-03-26T02:10:48.198601+0000","last_became_active":"2024-03-26T02:10:07.162329+0000","last_became_peered":"2024-03-26T02:10:07.162329+0000","last_unstale":"2024-03-26T02:10:48.198601+0000","last_undegraded":"2024-03-26T02:10:48.198601+0000","last_fullsized":"2024-03-26T02:10:48.198601+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:15:46.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347861+0000","last_change":"2024-03-26T02:10:03.147290+0000","last_active":"2024-03-26T02:10:31.347861+0000","last_peered":"2024-03-26T02:10:31.347861+0000","last_clean":"2024-03-26T02:10:31.347861+0000","last_became_active":"2024-03-26T02:10:03.145638+0000","last_became_peered":"2024-03-26T02:10:03.145638+0000","last_unstale":"2024-03-26T02:10:31.347861+0000","last_undegraded":"2024-03-26T02:10:31.347861+0000","last_fullsized":"2024-03-26T02:10:31.347861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:14:04.616380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345898+0000","last_change":"2024-03-26T02:10:05.153923+0000","last_active":"2024-03-26T02:10:31.345898+0000","last_peered":"2024-03-26T02:10:31.345898+0000","last_clean":"2024-03-26T02:10:31.345898+0000","last_became_active":"2024-03-26T02:10:05.153340+0000","last_became_peered":"2024-03-26T02:10:05.153340+0000","last_unstale":"2024-03-26T02:10:31.345898+0000","last_undegraded":"2024-03-26T02:10:31.345898+0000","last_fullsized":"2024-03-26T02:10:31.345898+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:16:22.378934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344324+0000","last_change":"2024-03-26T02:10:09.177959+0000","last_active":"2024-03-26T02:10:31.344324+0000","last_peered":"2024-03-26T02:10:31.344324+0000","last_clean":"2024-03-26T02:10:31.344324+0000","last_became_active":"2024-03-26T02:10:09.177595+0000","last_became_peered":"2024-03-26T02:10:09.177595+0000","last_unstale":"2024-03-26T02:10:31.344324+0000","last_undegraded":"2024-03-26T02:10:31.344324+0000","last_fullsized":"2024-03-26T02:10:31.344324+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:24:03.848583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345387+0000","last_change":"2024-03-26T02:10:07.157617+0000","last_active":"2024-03-26T02:10:31.345387+0000","last_peered":"2024-03-26T02:10:31.345387+0000","last_clean":"2024-03-26T02:10:31.345387+0000","last_became_active":"2024-03-26T02:10:07.156492+0000","last_became_peered":"2024-03-26T02:10:07.156492+0000","last_unstale":"2024-03-26T02:10:31.345387+0000","last_undegraded":"2024-03-26T02:10:31.345387+0000","last_fullsized":"2024-03-26T02:10:31.345387+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:33:15.838472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344686+0000","last_change":"2024-03-26T02:10:03.151316+0000","last_active":"2024-03-26T02:10:31.344686+0000","last_peered":"2024-03-26T02:10:31.344686+0000","last_clean":"2024-03-26T02:10:31.344686+0000","last_became_active":"2024-03-26T02:10:03.150852+0000","last_became_peered":"2024-03-26T02:10:03.150852+0000","last_unstale":"2024-03-26T02:10:31.344686+0000","last_undegraded":"2024-03-26T02:10:31.344686+0000","last_fullsized":"2024-03-26T02:10:31.344686+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:26:17.723569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343755+0000","last_change":"2024-03-26T02:10:05.152169+0000","last_active":"2024-03-26T02:10:31.343755+0000","last_peered":"2024-03-26T02:10:31.343755+0000","last_clean":"2024-03-26T02:10:31.343755+0000","last_became_active":"2024-03-26T02:10:05.151896+0000","last_became_peered":"2024-03-26T02:10:05.151896+0000","last_unstale":"2024-03-26T02:10:31.343755+0000","last_undegraded":"2024-03-26T02:10:31.343755+0000","last_fullsized":"2024-03-26T02:10:31.343755+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:23:34.403599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345130+0000","last_change":"2024-03-26T02:10:09.172840+0000","last_active":"2024-03-26T02:10:31.345130+0000","last_peered":"2024-03-26T02:10:31.345130+0000","last_clean":"2024-03-26T02:10:31.345130+0000","last_became_active":"2024-03-26T02:10:09.172503+0000","last_became_peered":"2024-03-26T02:10:09.172503+0000","last_unstale":"2024-03-26T02:10:31.345130+0000","last_undegraded":"2024-03-26T02:10:31.345130+0000","last_fullsized":"2024-03-26T02:10:31.345130+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:13:27.971276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344483+0000","last_change":"2024-03-26T02:10:07.157545+0000","last_active":"2024-03-26T02:10:31.344483+0000","last_peered":"2024-03-26T02:10:31.344483+0000","last_clean":"2024-03-26T02:10:31.344483+0000","last_became_active":"2024-03-26T02:10:07.157098+0000","last_became_peered":"2024-03-26T02:10:07.157098+0000","last_unstale":"2024-03-26T02:10:31.344483+0000","last_undegraded":"2024-03-26T02:10:31.344483+0000","last_fullsized":"2024-03-26T02:10:31.344483+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:51:51.549880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345869+0000","last_change":"2024-03-26T02:10:03.149259+0000","last_active":"2024-03-26T02:10:31.345869+0000","last_peered":"2024-03-26T02:10:31.345869+0000","last_clean":"2024-03-26T02:10:31.345869+0000","last_became_active":"2024-03-26T02:10:03.148528+0000","last_became_peered":"2024-03-26T02:10:03.148528+0000","last_unstale":"2024-03-26T02:10:31.345869+0000","last_undegraded":"2024-03-26T02:10:31.345869+0000","last_fullsized":"2024-03-26T02:10:31.345869+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:25.678733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346670+0000","last_change":"2024-03-26T02:10:05.148573+0000","last_active":"2024-03-26T02:10:31.346670+0000","last_peered":"2024-03-26T02:10:31.346670+0000","last_clean":"2024-03-26T02:10:31.346670+0000","last_became_active":"2024-03-26T02:10:05.148213+0000","last_became_peered":"2024-03-26T02:10:05.148213+0000","last_unstale":"2024-03-26T02:10:31.346670+0000","last_undegraded":"2024-03-26T02:10:31.346670+0000","last_fullsized":"2024-03-26T02:10:31.346670+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:21:51.711663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346319+0000","last_change":"2024-03-26T02:10:09.178594+0000","last_active":"2024-03-26T02:10:31.346319+0000","last_peered":"2024-03-26T02:10:31.346319+0000","last_clean":"2024-03-26T02:10:31.346319+0000","last_became_active":"2024-03-26T02:10:09.178342+0000","last_became_peered":"2024-03-26T02:10:09.178342+0000","last_unstale":"2024-03-26T02:10:31.346319+0000","last_undegraded":"2024-03-26T02:10:31.346319+0000","last_fullsized":"2024-03-26T02:10:31.346319+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:00:27.540660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348426+0000","last_change":"2024-03-26T02:10:03.146736+0000","last_active":"2024-03-26T02:10:31.348426+0000","last_peered":"2024-03-26T02:10:31.348426+0000","last_clean":"2024-03-26T02:10:31.348426+0000","last_became_active":"2024-03-26T02:10:03.146064+0000","last_became_peered":"2024-03-26T02:10:03.146064+0000","last_unstale":"2024-03-26T02:10:31.348426+0000","last_undegraded":"2024-03-26T02:10:31.348426+0000","last_fullsized":"2024-03-26T02:10:31.348426+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:56:26.049371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348039+0000","last_change":"2024-03-26T02:10:05.147570+0000","last_active":"2024-03-26T02:10:31.348039+0000","last_peered":"2024-03-26T02:10:31.348039+0000","last_clean":"2024-03-26T02:10:31.348039+0000","last_became_active":"2024-03-26T02:10:05.147225+0000","last_became_peered":"2024-03-26T02:10:05.147225+0000","last_unstale":"2024-03-26T02:10:31.348039+0000","last_undegraded":"2024-03-26T02:10:31.348039+0000","last_fullsized":"2024-03-26T02:10:31.348039+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:32:17.626514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:48.198663+0000","last_change":"2024-03-26T02:10:07.162899+0000","last_active":"2024-03-26T02:10:48.198663+0000","last_peered":"2024-03-26T02:10:48.198663+0000","last_clean":"2024-03-26T02:10:48.198663+0000","last_became_active":"2024-03-26T02:10:07.162496+0000","last_became_peered":"2024-03-26T02:10:07.162496+0000","last_unstale":"2024-03-26T02:10:48.198663+0000","last_undegraded":"2024-03-26T02:10:48.198663+0000","last_fullsized":"2024-03-26T02:10:48.198663+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:50:58.598137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342347+0000","last_change":"2024-03-26T02:10:09.174518+0000","last_active":"2024-03-26T02:10:31.342347+0000","last_peered":"2024-03-26T02:10:31.342347+0000","last_clean":"2024-03-26T02:10:31.342347+0000","last_became_active":"2024-03-26T02:10:09.174133+0000","last_became_peered":"2024-03-26T02:10:09.174133+0000","last_unstale":"2024-03-26T02:10:31.342347+0000","last_undegraded":"2024-03-26T02:10:31.342347+0000","last_fullsized":"2024-03-26T02:10:31.342347+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:28:22.095541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343814+0000","last_change":"2024-03-26T02:10:03.145727+0000","last_active":"2024-03-26T02:10:31.343814+0000","last_peered":"2024-03-26T02:10:31.343814+0000","last_clean":"2024-03-26T02:10:31.343814+0000","last_became_active":"2024-03-26T02:10:03.145184+0000","last_became_peered":"2024-03-26T02:10:03.145184+0000","last_unstale":"2024-03-26T02:10:31.343814+0000","last_undegraded":"2024-03-26T02:10:31.343814+0000","last_fullsized":"2024-03-26T02:10:31.343814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:03:28.975481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:48.198545+0000","last_change":"2024-03-26T02:10:05.149442+0000","last_active":"2024-03-26T02:10:48.198545+0000","last_peered":"2024-03-26T02:10:48.198545+0000","last_clean":"2024-03-26T02:10:48.198545+0000","last_became_active":"2024-03-26T02:10:05.149091+0000","last_became_peered":"2024-03-26T02:10:05.149091+0000","last_unstale":"2024-03-26T02:10:48.198545+0000","last_undegraded":"2024-03-26T02:10:48.198545+0000","last_fullsized":"2024-03-26T02:10:48.198545+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:04:31.262421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347544+0000","last_change":"2024-03-26T02:10:07.165732+0000","last_active":"2024-03-26T02:10:31.347544+0000","last_peered":"2024-03-26T02:10:31.347544+0000","last_clean":"2024-03-26T02:10:31.347544+0000","last_became_active":"2024-03-26T02:10:07.165431+0000","last_became_peered":"2024-03-26T02:10:07.165431+0000","last_unstale":"2024-03-26T02:10:31.347544+0000","last_undegraded":"2024-03-26T02:10:31.347544+0000","last_fullsized":"2024-03-26T02:10:31.347544+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:11:41.518529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343161+0000","last_change":"2024-03-26T02:10:09.173428+0000","last_active":"2024-03-26T02:10:31.343161+0000","last_peered":"2024-03-26T02:10:31.343161+0000","last_clean":"2024-03-26T02:10:31.343161+0000","last_became_active":"2024-03-26T02:10:09.173043+0000","last_became_peered":"2024-03-26T02:10:09.173043+0000","last_unstale":"2024-03-26T02:10:31.343161+0000","last_undegraded":"2024-03-26T02:10:31.343161+0000","last_fullsized":"2024-03-26T02:10:31.343161+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:34.318077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345020+0000","last_change":"2024-03-26T02:10:03.146247+0000","last_active":"2024-03-26T02:10:31.345020+0000","last_peered":"2024-03-26T02:10:31.345020+0000","last_clean":"2024-03-26T02:10:31.345020+0000","last_became_active":"2024-03-26T02:10:03.145765+0000","last_became_peered":"2024-03-26T02:10:03.145765+0000","last_unstale":"2024-03-26T02:10:31.345020+0000","last_undegraded":"2024-03-26T02:10:31.345020+0000","last_fullsized":"2024-03-26T02:10:31.345020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:21:14.687378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'27","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346697+0000","last_change":"2024-03-26T02:10:05.153864+0000","last_active":"2024-03-26T02:10:31.346697+0000","last_peered":"2024-03-26T02:10:31.346697+0000","last_clean":"2024-03-26T02:10:31.346697+0000","last_became_active":"2024-03-26T02:10:05.153304+0000","last_became_peered":"2024-03-26T02:10:05.153304+0000","last_unstale":"2024-03-26T02:10:31.346697+0000","last_undegraded":"2024-03-26T02:10:31.346697+0000","last_fullsized":"2024-03-26T02:10:31.346697+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:01:15.263692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343601+0000","last_change":"2024-03-26T02:10:07.157515+0000","last_active":"2024-03-26T02:10:31.343601+0000","last_peered":"2024-03-26T02:10:31.343601+0000","last_clean":"2024-03-26T02:10:31.343601+0000","last_became_active":"2024-03-26T02:10:07.155926+0000","last_became_peered":"2024-03-26T02:10:07.155926+0000","last_unstale":"2024-03-26T02:10:31.343601+0000","last_undegraded":"2024-03-26T02:10:31.343601+0000","last_fullsized":"2024-03-26T02:10:31.343601+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:32:03.677085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345158+0000","last_change":"2024-03-26T02:10:09.176291+0000","last_active":"2024-03-26T02:10:31.345158+0000","last_peered":"2024-03-26T02:10:31.345158+0000","last_clean":"2024-03-26T02:10:31.345158+0000","last_became_active":"2024-03-26T02:10:09.175411+0000","last_became_peered":"2024-03-26T02:10:09.175411+0000","last_unstale":"2024-03-26T02:10:31.345158+0000","last_undegraded":"2024-03-26T02:10:31.345158+0000","last_fullsized":"2024-03-26T02:10:31.345158+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:09:28.241894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345318+0000","last_change":"2024-03-26T02:10:05.148986+0000","last_active":"2024-03-26T02:10:31.345318+0000","last_peered":"2024-03-26T02:10:31.345318+0000","last_clean":"2024-03-26T02:10:31.345318+0000","last_became_active":"2024-03-26T02:10:05.148293+0000","last_became_peered":"2024-03-26T02:10:05.148293+0000","last_unstale":"2024-03-26T02:10:31.345318+0000","last_undegraded":"2024-03-26T02:10:31.345318+0000","last_fullsized":"2024-03-26T02:10:31.345318+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:41:41.381981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342783+0000","last_change":"2024-03-26T02:10:03.146220+0000","last_active":"2024-03-26T02:10:31.342783+0000","last_peered":"2024-03-26T02:10:31.342783+0000","last_clean":"2024-03-26T02:10:31.342783+0000","last_became_active":"2024-03-26T02:10:03.145318+0000","last_became_peered":"2024-03-26T02:10:03.145318+0000","last_unstale":"2024-03-26T02:10:31.342783+0000","last_undegraded":"2024-03-26T02:10:31.342783+0000","last_fullsized":"2024-03-26T02:10:31.342783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:05:38.180067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345699+0000","last_change":"2024-03-26T02:10:07.158901+0000","last_active":"2024-03-26T02:10:31.345699+0000","last_peered":"2024-03-26T02:10:31.345699+0000","last_clean":"2024-03-26T02:10:31.345699+0000","last_became_active":"2024-03-26T02:10:07.158572+0000","last_became_peered":"2024-03-26T02:10:07.158572+0000","last_unstale":"2024-03-26T02:10:31.345699+0000","last_undegraded":"2024-03-26T02:10:31.345699+0000","last_fullsized":"2024-03-26T02:10:31.345699+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:20:02.303605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346969+0000","last_change":"2024-03-26T02:10:09.172226+0000","last_active":"2024-03-26T02:10:31.346969+0000","last_peered":"2024-03-26T02:10:31.346969+0000","last_clean":"2024-03-26T02:10:31.346969+0000","last_became_active":"2024-03-26T02:10:09.171923+0000","last_became_peered":"2024-03-26T02:10:09.171923+0000","last_unstale":"2024-03-26T02:10:31.346969+0000","last_undegraded":"2024-03-26T02:10:31.346969+0000","last_fullsized":"2024-03-26T02:10:31.346969+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:48.642310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342285+0000","last_change":"2024-03-26T02:10:05.148018+0000","last_active":"2024-03-26T02:10:31.342285+0000","last_peered":"2024-03-26T02:10:31.342285+0000","last_clean":"2024-03-26T02:10:31.342285+0000","last_became_active":"2024-03-26T02:10:05.147177+0000","last_became_peered":"2024-03-26T02:10:05.147177+0000","last_unstale":"2024-03-26T02:10:31.342285+0000","last_undegraded":"2024-03-26T02:10:31.342285+0000","last_fullsized":"2024-03-26T02:10:31.342285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:32:39.071397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345781+0000","last_change":"2024-03-26T02:10:03.147832+0000","last_active":"2024-03-26T02:10:31.345781+0000","last_peered":"2024-03-26T02:10:31.345781+0000","last_clean":"2024-03-26T02:10:31.345781+0000","last_became_active":"2024-03-26T02:10:03.146409+0000","last_became_peered":"2024-03-26T02:10:03.146409+0000","last_unstale":"2024-03-26T02:10:31.345781+0000","last_undegraded":"2024-03-26T02:10:31.345781+0000","last_fullsized":"2024-03-26T02:10:31.345781+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:48:13.557087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:33.457254+0000","last_change":"2024-03-26T02:09:52.036121+0000","last_active":"2024-03-26T02:10:33.457254+0000","last_peered":"2024-03-26T02:10:33.457254+0000","last_clean":"2024-03-26T02:10:33.457254+0000","last_became_active":"2024-03-26T02:09:52.035795+0000","last_became_peered":"2024-03-26T02:09:52.035795+0000","last_unstale":"2024-03-26T02:10:33.457254+0000","last_undegraded":"2024-03-26T02:10:33.457254+0000","last_fullsized":"2024-03-26T02:10:33.457254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:09:51.029144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:09:51.029144+0000","last_clean_scrub_stamp":"2024-03-26T02:09:51.029144+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:49:08.163765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346756+0000","last_change":"2024-03-26T02:10:07.157775+0000","last_active":"2024-03-26T02:10:31.346756+0000","last_peered":"2024-03-26T02:10:31.346756+0000","last_clean":"2024-03-26T02:10:31.346756+0000","last_became_active":"2024-03-26T02:10:07.157358+0000","last_became_peered":"2024-03-26T02:10:07.157358+0000","last_unstale":"2024-03-26T02:10:31.346756+0000","last_undegraded":"2024-03-26T02:10:31.346756+0000","last_fullsized":"2024-03-26T02:10:31.346756+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:08.392906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345051+0000","last_change":"2024-03-26T02:10:09.176510+0000","last_active":"2024-03-26T02:10:31.345051+0000","last_peered":"2024-03-26T02:10:31.345051+0000","last_clean":"2024-03-26T02:10:31.345051+0000","last_became_active":"2024-03-26T02:10:09.174022+0000","last_became_peered":"2024-03-26T02:10:09.174022+0000","last_unstale":"2024-03-26T02:10:31.345051+0000","last_undegraded":"2024-03-26T02:10:31.345051+0000","last_fullsized":"2024-03-26T02:10:31.345051+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:47:58.808062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344820+0000","last_change":"2024-03-26T02:10:05.151043+0000","last_active":"2024-03-26T02:10:31.344820+0000","last_peered":"2024-03-26T02:10:31.344820+0000","last_clean":"2024-03-26T02:10:31.344820+0000","last_became_active":"2024-03-26T02:10:05.150725+0000","last_became_peered":"2024-03-26T02:10:05.150725+0000","last_unstale":"2024-03-26T02:10:31.344820+0000","last_undegraded":"2024-03-26T02:10:31.344820+0000","last_fullsized":"2024-03-26T02:10:31.344820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:29:12.690541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343878+0000","last_change":"2024-03-26T02:10:03.148586+0000","last_active":"2024-03-26T02:10:31.343878+0000","last_peered":"2024-03-26T02:10:31.343878+0000","last_clean":"2024-03-26T02:10:31.343878+0000","last_became_active":"2024-03-26T02:10:03.148037+0000","last_became_peered":"2024-03-26T02:10:03.148037+0000","last_unstale":"2024-03-26T02:10:31.343878+0000","last_undegraded":"2024-03-26T02:10:31.343878+0000","last_fullsized":"2024-03-26T02:10:31.343878+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:48:20.731637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347581+0000","last_change":"2024-03-26T02:10:07.165653+0000","last_active":"2024-03-26T02:10:31.347581+0000","last_peered":"2024-03-26T02:10:31.347581+0000","last_clean":"2024-03-26T02:10:31.347581+0000","last_became_active":"2024-03-26T02:10:07.165130+0000","last_became_peered":"2024-03-26T02:10:07.165130+0000","last_unstale":"2024-03-26T02:10:31.347581+0000","last_undegraded":"2024-03-26T02:10:31.347581+0000","last_fullsized":"2024-03-26T02:10:31.347581+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:09:10.434457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345113+0000","last_change":"2024-03-26T02:10:09.176720+0000","last_active":"2024-03-26T02:10:31.345113+0000","last_peered":"2024-03-26T02:10:31.345113+0000","last_clean":"2024-03-26T02:10:31.345113+0000","last_became_active":"2024-03-26T02:10:09.176269+0000","last_became_peered":"2024-03-26T02:10:09.176269+0000","last_unstale":"2024-03-26T02:10:31.345113+0000","last_undegraded":"2024-03-26T02:10:31.345113+0000","last_fullsized":"2024-03-26T02:10:31.345113+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:59:41.438490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348362+0000","last_change":"2024-03-26T02:10:03.148495+0000","last_active":"2024-03-26T02:10:31.348362+0000","last_peered":"2024-03-26T02:10:31.348362+0000","last_clean":"2024-03-26T02:10:31.348362+0000","last_became_active":"2024-03-26T02:10:03.147232+0000","last_became_peered":"2024-03-26T02:10:03.147232+0000","last_unstale":"2024-03-26T02:10:31.348362+0000","last_undegraded":"2024-03-26T02:10:31.348362+0000","last_fullsized":"2024-03-26T02:10:31.348362+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:36:27.990383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'34","reported_seq":94,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:48.198389+0000","last_change":"2024-03-26T02:10:05.150785+0000","last_active":"2024-03-26T02:10:48.198389+0000","last_peered":"2024-03-26T02:10:48.198389+0000","last_clean":"2024-03-26T02:10:48.198389+0000","last_became_active":"2024-03-26T02:10:05.150409+0000","last_became_peered":"2024-03-26T02:10:05.150409+0000","last_unstale":"2024-03-26T02:10:48.198389+0000","last_undegraded":"2024-03-26T02:10:48.198389+0000","last_fullsized":"2024-03-26T02:10:48.198389+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:47:18.936093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:48.198628+0000","last_change":"2024-03-26T02:10:07.162869+0000","last_active":"2024-03-26T02:10:48.198628+0000","last_peered":"2024-03-26T02:10:48.198628+0000","last_clean":"2024-03-26T02:10:48.198628+0000","last_became_active":"2024-03-26T02:10:07.162526+0000","last_became_peered":"2024-03-26T02:10:07.162526+0000","last_unstale":"2024-03-26T02:10:48.198628+0000","last_undegraded":"2024-03-26T02:10:48.198628+0000","last_fullsized":"2024-03-26T02:10:48.198628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:37:12.542294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344176+0000","last_change":"2024-03-26T02:10:09.178302+0000","last_active":"2024-03-26T02:10:31.344176+0000","last_peered":"2024-03-26T02:10:31.344176+0000","last_clean":"2024-03-26T02:10:31.344176+0000","last_became_active":"2024-03-26T02:10:09.177853+0000","last_became_peered":"2024-03-26T02:10:09.177853+0000","last_unstale":"2024-03-26T02:10:31.344176+0000","last_undegraded":"2024-03-26T02:10:31.344176+0000","last_fullsized":"2024-03-26T02:10:31.344176+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:28:22.954793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344789+0000","last_change":"2024-03-26T02:10:03.146268+0000","last_active":"2024-03-26T02:10:31.344789+0000","last_peered":"2024-03-26T02:10:31.344789+0000","last_clean":"2024-03-26T02:10:31.344789+0000","last_became_active":"2024-03-26T02:10:03.145238+0000","last_became_peered":"2024-03-26T02:10:03.145238+0000","last_unstale":"2024-03-26T02:10:31.344789+0000","last_undegraded":"2024-03-26T02:10:31.344789+0000","last_fullsized":"2024-03-26T02:10:31.344789+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:58:29.291845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345886+0000","last_change":"2024-03-26T02:10:05.152874+0000","last_active":"2024-03-26T02:10:31.345886+0000","last_peered":"2024-03-26T02:10:31.345886+0000","last_clean":"2024-03-26T02:10:31.345886+0000","last_became_active":"2024-03-26T02:10:05.152569+0000","last_became_peered":"2024-03-26T02:10:05.152569+0000","last_unstale":"2024-03-26T02:10:31.345886+0000","last_undegraded":"2024-03-26T02:10:31.345886+0000","last_fullsized":"2024-03-26T02:10:31.345886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:58.887192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:48.198407+0000","last_change":"2024-03-26T02:10:07.159627+0000","last_active":"2024-03-26T02:10:48.198407+0000","last_peered":"2024-03-26T02:10:48.198407+0000","last_clean":"2024-03-26T02:10:48.198407+0000","last_became_active":"2024-03-26T02:10:07.159169+0000","last_became_peered":"2024-03-26T02:10:07.159169+0000","last_unstale":"2024-03-26T02:10:48.198407+0000","last_undegraded":"2024-03-26T02:10:48.198407+0000","last_fullsized":"2024-03-26T02:10:48.198407+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:04:28.857380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346311+0000","last_change":"2024-03-26T02:10:09.178722+0000","last_active":"2024-03-26T02:10:31.346311+0000","last_peered":"2024-03-26T02:10:31.346311+0000","last_clean":"2024-03-26T02:10:31.346311+0000","last_became_active":"2024-03-26T02:10:09.178445+0000","last_became_peered":"2024-03-26T02:10:09.178445+0000","last_unstale":"2024-03-26T02:10:31.346311+0000","last_undegraded":"2024-03-26T02:10:31.346311+0000","last_fullsized":"2024-03-26T02:10:31.346311+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:21:42.960113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345457+0000","last_change":"2024-03-26T02:10:03.146592+0000","last_active":"2024-03-26T02:10:31.345457+0000","last_peered":"2024-03-26T02:10:31.345457+0000","last_clean":"2024-03-26T02:10:31.345457+0000","last_became_active":"2024-03-26T02:10:03.146186+0000","last_became_peered":"2024-03-26T02:10:03.146186+0000","last_unstale":"2024-03-26T02:10:31.345457+0000","last_undegraded":"2024-03-26T02:10:31.345457+0000","last_fullsized":"2024-03-26T02:10:31.345457+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:33:24.975689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344941+0000","last_change":"2024-03-26T02:10:05.152525+0000","last_active":"2024-03-26T02:10:31.344941+0000","last_peered":"2024-03-26T02:10:31.344941+0000","last_clean":"2024-03-26T02:10:31.344941+0000","last_became_active":"2024-03-26T02:10:05.151932+0000","last_became_peered":"2024-03-26T02:10:05.151932+0000","last_unstale":"2024-03-26T02:10:31.344941+0000","last_undegraded":"2024-03-26T02:10:31.344941+0000","last_fullsized":"2024-03-26T02:10:31.344941+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:44:37.133138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346429+0000","last_change":"2024-03-26T02:10:07.163466+0000","last_active":"2024-03-26T02:10:31.346429+0000","last_peered":"2024-03-26T02:10:31.346429+0000","last_clean":"2024-03-26T02:10:31.346429+0000","last_became_active":"2024-03-26T02:10:07.163179+0000","last_became_peered":"2024-03-26T02:10:07.163179+0000","last_unstale":"2024-03-26T02:10:31.346429+0000","last_undegraded":"2024-03-26T02:10:31.346429+0000","last_fullsized":"2024-03-26T02:10:31.346429+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:47:55.366104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344881+0000","last_change":"2024-03-26T02:10:09.176557+0000","last_active":"2024-03-26T02:10:31.344881+0000","last_peered":"2024-03-26T02:10:31.344881+0000","last_clean":"2024-03-26T02:10:31.344881+0000","last_became_active":"2024-03-26T02:10:09.172801+0000","last_became_peered":"2024-03-26T02:10:09.172801+0000","last_unstale":"2024-03-26T02:10:31.344881+0000","last_undegraded":"2024-03-26T02:10:31.344881+0000","last_fullsized":"2024-03-26T02:10:31.344881+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:57:55.215601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:48.198577+0000","last_change":"2024-03-26T02:10:07.155357+0000","last_active":"2024-03-26T02:10:48.198577+0000","last_peered":"2024-03-26T02:10:48.198577+0000","last_clean":"2024-03-26T02:10:48.198577+0000","last_became_active":"2024-03-26T02:10:07.155032+0000","last_became_peered":"2024-03-26T02:10:07.155032+0000","last_unstale":"2024-03-26T02:10:48.198577+0000","last_undegraded":"2024-03-26T02:10:48.198577+0000","last_fullsized":"2024-03-26T02:10:48.198577+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:20:25.573344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344000+0000","last_change":"2024-03-26T02:10:03.147704+0000","last_active":"2024-03-26T02:10:31.344000+0000","last_peered":"2024-03-26T02:10:31.344000+0000","last_clean":"2024-03-26T02:10:31.344000+0000","last_became_active":"2024-03-26T02:10:03.145871+0000","last_became_peered":"2024-03-26T02:10:03.145871+0000","last_unstale":"2024-03-26T02:10:31.344000+0000","last_undegraded":"2024-03-26T02:10:31.344000+0000","last_fullsized":"2024-03-26T02:10:31.344000+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:13:18.131271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346525+0000","last_change":"2024-03-26T02:10:05.148110+0000","last_active":"2024-03-26T02:10:31.346525+0000","last_peered":"2024-03-26T02:10:31.346525+0000","last_clean":"2024-03-26T02:10:31.346525+0000","last_became_active":"2024-03-26T02:10:05.147014+0000","last_became_peered":"2024-03-26T02:10:05.147014+0000","last_unstale":"2024-03-26T02:10:31.346525+0000","last_undegraded":"2024-03-26T02:10:31.346525+0000","last_fullsized":"2024-03-26T02:10:31.346525+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:52:09.112994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347711+0000","last_change":"2024-03-26T02:10:09.174800+0000","last_active":"2024-03-26T02:10:31.347711+0000","last_peered":"2024-03-26T02:10:31.347711+0000","last_clean":"2024-03-26T02:10:31.347711+0000","last_became_active":"2024-03-26T02:10:09.174506+0000","last_became_peered":"2024-03-26T02:10:09.174506+0000","last_unstale":"2024-03-26T02:10:31.347711+0000","last_undegraded":"2024-03-26T02:10:31.347711+0000","last_fullsized":"2024-03-26T02:10:31.347711+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:05:27.786474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345462+0000","last_change":"2024-03-26T02:10:07.157389+0000","last_active":"2024-03-26T02:10:31.345462+0000","last_peered":"2024-03-26T02:10:31.345462+0000","last_clean":"2024-03-26T02:10:31.345462+0000","last_became_active":"2024-03-26T02:10:07.156901+0000","last_became_peered":"2024-03-26T02:10:07.156901+0000","last_unstale":"2024-03-26T02:10:31.345462+0000","last_undegraded":"2024-03-26T02:10:31.345462+0000","last_fullsized":"2024-03-26T02:10:31.345462+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:00:53.748792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342615+0000","last_change":"2024-03-26T02:10:03.147000+0000","last_active":"2024-03-26T02:10:31.342615+0000","last_peered":"2024-03-26T02:10:31.342615+0000","last_clean":"2024-03-26T02:10:31.342615+0000","last_became_active":"2024-03-26T02:10:03.146464+0000","last_became_peered":"2024-03-26T02:10:03.146464+0000","last_unstale":"2024-03-26T02:10:31.342615+0000","last_undegraded":"2024-03-26T02:10:31.342615+0000","last_fullsized":"2024-03-26T02:10:31.342615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:53:09.795049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345966+0000","last_change":"2024-03-26T02:10:05.148745+0000","last_active":"2024-03-26T02:10:31.345966+0000","last_peered":"2024-03-26T02:10:31.345966+0000","last_clean":"2024-03-26T02:10:31.345966+0000","last_became_active":"2024-03-26T02:10:05.147575+0000","last_became_peered":"2024-03-26T02:10:05.147575+0000","last_unstale":"2024-03-26T02:10:31.345966+0000","last_undegraded":"2024-03-26T02:10:31.345966+0000","last_fullsized":"2024-03-26T02:10:31.345966+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:18:24.605202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347842+0000","last_change":"2024-03-26T02:10:09.171392+0000","last_active":"2024-03-26T02:10:31.347842+0000","last_peered":"2024-03-26T02:10:31.347842+0000","last_clean":"2024-03-26T02:10:31.347842+0000","last_became_active":"2024-03-26T02:10:09.171063+0000","last_became_peered":"2024-03-26T02:10:09.171063+0000","last_unstale":"2024-03-26T02:10:31.347842+0000","last_undegraded":"2024-03-26T02:10:31.347842+0000","last_fullsized":"2024-03-26T02:10:31.347842+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:01:01.822547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348113+0000","last_change":"2024-03-26T02:10:03.149001+0000","last_active":"2024-03-26T02:10:31.348113+0000","last_peered":"2024-03-26T02:10:31.348113+0000","last_clean":"2024-03-26T02:10:31.348113+0000","last_became_active":"2024-03-26T02:10:03.148093+0000","last_became_peered":"2024-03-26T02:10:03.148093+0000","last_unstale":"2024-03-26T02:10:31.348113+0000","last_undegraded":"2024-03-26T02:10:31.348113+0000","last_fullsized":"2024-03-26T02:10:31.348113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:51:55.430328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342889+0000","last_change":"2024-03-26T02:10:05.147551+0000","last_active":"2024-03-26T02:10:31.342889+0000","last_peered":"2024-03-26T02:10:31.342889+0000","last_clean":"2024-03-26T02:10:31.342889+0000","last_became_active":"2024-03-26T02:10:05.147194+0000","last_became_peered":"2024-03-26T02:10:05.147194+0000","last_unstale":"2024-03-26T02:10:31.342889+0000","last_undegraded":"2024-03-26T02:10:31.342889+0000","last_fullsized":"2024-03-26T02:10:31.342889+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:07:22.685556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345694+0000","last_change":"2024-03-26T02:10:07.154700+0000","last_active":"2024-03-26T02:10:31.345694+0000","last_peered":"2024-03-26T02:10:31.345694+0000","last_clean":"2024-03-26T02:10:31.345694+0000","last_became_active":"2024-03-26T02:10:07.154438+0000","last_became_peered":"2024-03-26T02:10:07.154438+0000","last_unstale":"2024-03-26T02:10:31.345694+0000","last_undegraded":"2024-03-26T02:10:31.345694+0000","last_fullsized":"2024-03-26T02:10:31.345694+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:23:17.868509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343243+0000","last_change":"2024-03-26T02:10:09.174105+0000","last_active":"2024-03-26T02:10:31.343243+0000","last_peered":"2024-03-26T02:10:31.343243+0000","last_clean":"2024-03-26T02:10:31.343243+0000","last_became_active":"2024-03-26T02:10:09.173440+0000","last_became_peered":"2024-03-26T02:10:09.173440+0000","last_unstale":"2024-03-26T02:10:31.343243+0000","last_undegraded":"2024-03-26T02:10:31.343243+0000","last_fullsized":"2024-03-26T02:10:31.343243+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:00:59.907859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:48.198525+0000","last_change":"2024-03-26T02:10:07.157698+0000","last_active":"2024-03-26T02:10:48.198525+0000","last_peered":"2024-03-26T02:10:48.198525+0000","last_clean":"2024-03-26T02:10:48.198525+0000","last_became_active":"2024-03-26T02:10:07.157349+0000","last_became_peered":"2024-03-26T02:10:07.157349+0000","last_unstale":"2024-03-26T02:10:48.198525+0000","last_undegraded":"2024-03-26T02:10:48.198525+0000","last_fullsized":"2024-03-26T02:10:48.198525+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:12:01.502132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346082+0000","last_change":"2024-03-26T02:10:03.148086+0000","last_active":"2024-03-26T02:10:31.346082+0000","last_peered":"2024-03-26T02:10:31.346082+0000","last_clean":"2024-03-26T02:10:31.346082+0000","last_became_active":"2024-03-26T02:10:03.147305+0000","last_became_peered":"2024-03-26T02:10:03.147305+0000","last_unstale":"2024-03-26T02:10:31.346082+0000","last_undegraded":"2024-03-26T02:10:31.346082+0000","last_fullsized":"2024-03-26T02:10:31.346082+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:55:57.277558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'26","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345334+0000","last_change":"2024-03-26T02:10:05.149623+0000","last_active":"2024-03-26T02:10:31.345334+0000","last_peered":"2024-03-26T02:10:31.345334+0000","last_clean":"2024-03-26T02:10:31.345334+0000","last_became_active":"2024-03-26T02:10:05.148175+0000","last_became_peered":"2024-03-26T02:10:05.148175+0000","last_unstale":"2024-03-26T02:10:31.345334+0000","last_undegraded":"2024-03-26T02:10:31.345334+0000","last_fullsized":"2024-03-26T02:10:31.345334+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:42:15.868815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345716+0000","last_change":"2024-03-26T02:10:09.174960+0000","last_active":"2024-03-26T02:10:31.345716+0000","last_peered":"2024-03-26T02:10:31.345716+0000","last_clean":"2024-03-26T02:10:31.345716+0000","last_became_active":"2024-03-26T02:10:09.174576+0000","last_became_peered":"2024-03-26T02:10:09.174576+0000","last_unstale":"2024-03-26T02:10:31.345716+0000","last_undegraded":"2024-03-26T02:10:31.345716+0000","last_fullsized":"2024-03-26T02:10:31.345716+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:53:29.187511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347568+0000","last_change":"2024-03-26T02:10:03.146340+0000","last_active":"2024-03-26T02:10:31.347568+0000","last_peered":"2024-03-26T02:10:31.347568+0000","last_clean":"2024-03-26T02:10:31.347568+0000","last_became_active":"2024-03-26T02:10:03.145926+0000","last_became_peered":"2024-03-26T02:10:03.145926+0000","last_unstale":"2024-03-26T02:10:31.347568+0000","last_undegraded":"2024-03-26T02:10:31.347568+0000","last_fullsized":"2024-03-26T02:10:31.347568+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:37:44.880105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.347039+0000","last_change":"2024-03-26T02:10:05.149648+0000","last_active":"2024-03-26T02:10:31.347039+0000","last_peered":"2024-03-26T02:10:31.347039+0000","last_clean":"2024-03-26T02:10:31.347039+0000","last_became_active":"2024-03-26T02:10:05.149205+0000","last_became_peered":"2024-03-26T02:10:05.149205+0000","last_unstale":"2024-03-26T02:10:31.347039+0000","last_undegraded":"2024-03-26T02:10:31.347039+0000","last_fullsized":"2024-03-26T02:10:31.347039+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:54:37.358709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344811+0000","last_change":"2024-03-26T02:10:07.157966+0000","last_active":"2024-03-26T02:10:31.344811+0000","last_peered":"2024-03-26T02:10:31.344811+0000","last_clean":"2024-03-26T02:10:31.344811+0000","last_became_active":"2024-03-26T02:10:07.157233+0000","last_became_peered":"2024-03-26T02:10:07.157233+0000","last_unstale":"2024-03-26T02:10:31.344811+0000","last_undegraded":"2024-03-26T02:10:31.344811+0000","last_fullsized":"2024-03-26T02:10:31.344811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:13:23.456041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.341836+0000","last_change":"2024-03-26T02:10:09.172816+0000","last_active":"2024-03-26T02:10:31.341836+0000","last_peered":"2024-03-26T02:10:31.341836+0000","last_clean":"2024-03-26T02:10:31.341836+0000","last_became_active":"2024-03-26T02:10:09.172445+0000","last_became_peered":"2024-03-26T02:10:09.172445+0000","last_unstale":"2024-03-26T02:10:31.341836+0000","last_undegraded":"2024-03-26T02:10:31.341836+0000","last_fullsized":"2024-03-26T02:10:31.341836+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:02:22.260705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.348186+0000","last_change":"2024-03-26T02:10:03.147972+0000","last_active":"2024-03-26T02:10:31.348186+0000","last_peered":"2024-03-26T02:10:31.348186+0000","last_clean":"2024-03-26T02:10:31.348186+0000","last_became_active":"2024-03-26T02:10:03.147224+0000","last_became_peered":"2024-03-26T02:10:03.147224+0000","last_unstale":"2024-03-26T02:10:31.348186+0000","last_undegraded":"2024-03-26T02:10:31.348186+0000","last_fullsized":"2024-03-26T02:10:31.348186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:42:27.433852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346767+0000","last_change":"2024-03-26T02:10:05.150266+0000","last_active":"2024-03-26T02:10:31.346767+0000","last_peered":"2024-03-26T02:10:31.346767+0000","last_clean":"2024-03-26T02:10:31.346767+0000","last_became_active":"2024-03-26T02:10:05.149898+0000","last_became_peered":"2024-03-26T02:10:05.149898+0000","last_unstale":"2024-03-26T02:10:31.346767+0000","last_undegraded":"2024-03-26T02:10:31.346767+0000","last_fullsized":"2024-03-26T02:10:31.346767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:36:39.044417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.342574+0000","last_change":"2024-03-26T02:10:07.160503+0000","last_active":"2024-03-26T02:10:31.342574+0000","last_peered":"2024-03-26T02:10:31.342574+0000","last_clean":"2024-03-26T02:10:31.342574+0000","last_became_active":"2024-03-26T02:10:07.159992+0000","last_became_peered":"2024-03-26T02:10:07.159992+0000","last_unstale":"2024-03-26T02:10:31.342574+0000","last_undegraded":"2024-03-26T02:10:31.342574+0000","last_fullsized":"2024-03-26T02:10:31.342574+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:55:54.267028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.343971+0000","last_change":"2024-03-26T02:10:09.176711+0000","last_active":"2024-03-26T02:10:31.343971+0000","last_peered":"2024-03-26T02:10:31.343971+0000","last_clean":"2024-03-26T02:10:31.343971+0000","last_became_active":"2024-03-26T02:10:09.176345+0000","last_became_peered":"2024-03-26T02:10:09.176345+0000","last_unstale":"2024-03-26T02:10:31.343971+0000","last_undegraded":"2024-03-26T02:10:31.343971+0000","last_fullsized":"2024-03-26T02:10:31.343971+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:08.155011+0000","last_clean_scrub_stamp":"2024-03-26T02:10:08.155011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:07.347065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.344615+0000","last_change":"2024-03-26T02:10:03.147091+0000","last_active":"2024-03-26T02:10:31.344615+0000","last_peered":"2024-03-26T02:10:31.344615+0000","last_clean":"2024-03-26T02:10:31.344615+0000","last_became_active":"2024-03-26T02:10:03.146748+0000","last_became_peered":"2024-03-26T02:10:03.146748+0000","last_unstale":"2024-03-26T02:10:31.344615+0000","last_undegraded":"2024-03-26T02:10:31.344615+0000","last_fullsized":"2024-03-26T02:10:31.344615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:02.132542+0000","last_clean_scrub_stamp":"2024-03-26T02:10:02.132542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:36:40.299573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.345542+0000","last_change":"2024-03-26T02:10:05.149723+0000","last_active":"2024-03-26T02:10:31.345542+0000","last_peered":"2024-03-26T02:10:31.345542+0000","last_clean":"2024-03-26T02:10:31.345542+0000","last_became_active":"2024-03-26T02:10:05.149266+0000","last_became_peered":"2024-03-26T02:10:05.149266+0000","last_unstale":"2024-03-26T02:10:31.345542+0000","last_undegraded":"2024-03-26T02:10:31.345542+0000","last_fullsized":"2024-03-26T02:10:31.345542+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:04.135748+0000","last_clean_scrub_stamp":"2024-03-26T02:10:04.135748+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:03:08.495256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-26T02:10:31.346167+0000","last_change":"2024-03-26T02:10:07.164844+0000","last_active":"2024-03-26T02:10:31.346167+0000","last_peered":"2024-03-26T02:10:31.346167+0000","last_clean":"2024-03-26T02:10:31.346167+0000","last_became_active":"2024-03-26T02:10:07.163700+0000","last_became_peered":"2024-03-26T02:10:07.163700+0000","last_unstale":"2024-03-26T02:10:31.346167+0000","last_undegraded":"2024-03-26T02:10:31.346167+0000","last_fullsized":"2024-03-26T02:10:31.346167+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:10:06.139453+0000","last_clean_scrub_stamp":"2024-03-26T02:10:06.139453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:11:09.664227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":589,"ondisk_log_size":589,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411344,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738395,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T02:11:02.462 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T02:11:02.462 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T02:11:02.462 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T02:11:02.463 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T02:11:02.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:02 smithi052 ceph-mon[25889]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:02.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:02 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:02.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:02 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:02.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:02 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:03.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:02 smithi165 ceph-mon[27627]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:03.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:02 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:03.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:02 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:03.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:02 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:03.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:02 smithi191 ceph-mon[27701]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:03.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:02 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:03.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:02 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:03.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:02 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:03.366 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:11:04.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:04 smithi052 ceph-mon[25889]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:04.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:04 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:04.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:04 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:04.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:04 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:04.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:04 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:04 smithi191 ceph-mon[27701]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:04 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:04.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:04 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:04 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:04.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:04 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:04.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:04 smithi165 ceph-mon[27627]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:04.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:04 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:04.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:04 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:04.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:04 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:04.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:04 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:05.896 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-26T02:11:05.896 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T02:11:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:06 smithi191 ceph-mon[27701]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:06.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:06 smithi191 ceph-mon[27701]: from='client.? 172.21.15.52:0/1972188055' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T02:11:06.663 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:06 smithi052 ceph-mon[25889]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:06.663 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:06 smithi052 ceph-mon[25889]: from='client.? 172.21.15.52:0/1972188055' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T02:11:06.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:06 smithi165 ceph-mon[27627]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:06.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:06 smithi165 ceph-mon[27627]: from='client.? 172.21.15.52:0/1972188055' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T02:11:07.015 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T02:11:07.016 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T02:11:07.016 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T02:11:07.026 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-03-26T02:11:07.027 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-26T02:11:07.268 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:11:07.641 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:07 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ::ffff:172.21.15.165 - - [26/Mar/2024:02:11:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:11:07.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:07 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:07.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:07 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:07.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:07 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:07.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:07 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:07.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:07 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:07.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:07 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:07 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:07 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:08.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:07 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:08.667 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:08 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[49420]: ts=2024-03-26T02:11:08.447Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00227107s 2024-03-26T02:11:08.905 INFO:teuthology.orchestra.run.smithi052.stdout:167 167 2024-03-26T02:11:08.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:08 smithi165 ceph-mon[27627]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:08.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:08 smithi052 ceph-mon[25889]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:09.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:08 smithi191 ceph-mon[27701]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:09.666 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-26T02:11:09.909 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:11:10.795 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:10 smithi165 ceph-mon[27627]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:10.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:10 smithi052 ceph-mon[25889]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:11.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:10 smithi191 ceph-mon[27701]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:11.828 INFO:teuthology.orchestra.run.smithi052.stdout:Backend: cephadm 2024-03-26T02:11:11.828 INFO:teuthology.orchestra.run.smithi052.stdout:Available: Yes 2024-03-26T02:11:11.828 INFO:teuthology.orchestra.run.smithi052.stdout:Paused: No 2024-03-26T02:11:12.419 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-26T02:11:12.659 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:11:12.686 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:12 smithi165 systemd[1]: Starting Ceph grafana.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:12.937 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:12 smithi165 ceph-mon[27627]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:12.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:12 smithi165 ceph-mon[27627]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:12.938 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:12 smithi165 podman[38648]: 2024-03-26 02:11:12.692256662 +0000 UTC m=+0.017301270 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-26T02:11:12.938 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:12 smithi165 podman[38648]: 2024-03-26 02:11:12.806581262 +0000 UTC m=+0.131625911 container create cb607c912b0472841df6abc63207ab4b1a73591a421e14452d6d8f8357a36786 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, vcs-type=git, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, name=ubi8) 2024-03-26T02:11:12.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:12 smithi052 ceph-mon[25889]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:12.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:12 smithi052 ceph-mon[25889]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:13.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:12 smithi191 ceph-mon[27701]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:13.037 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:12 smithi191 ceph-mon[27701]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:13.309 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 podman[38648]: 2024-03-26 02:11:13.058772145 +0000 UTC m=+0.383816753 container init cb607c912b0472841df6abc63207ab4b1a73591a421e14452d6d8f8357a36786 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a, version=8.5, vendor=Red Hat, Inc., distribution-scope=public, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, architecture=x86_64, release=239.1651231664, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-26T02:11:13.309 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 podman[38648]: 2024-03-26 02:11:13.064032825 +0000 UTC m=+0.389077433 container start cb607c912b0472841df6abc63207ab4b1a73591a421e14452d6d8f8357a36786 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, distribution-scope=public, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.openshift.expose-services=, release=239.1651231664, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, vcs-type=git) 2024-03-26T02:11:13.309 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 bash[38648]: cb607c912b0472841df6abc63207ab4b1a73591a421e14452d6d8f8357a36786 2024-03-26T02:11:13.310 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 systemd[1]: Started Ceph grafana.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:11:13.310 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218480674Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T02:11:13.310 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.21870431Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T02:11:13.310 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218730516Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T02:11:13.310 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218745167Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T02:11:13.310 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218757419Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T02:11:13.310 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218771525Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T02:11:13.310 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218781774Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T02:11:13.310 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218792269Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T02:11:13.310 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218802777Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218813144Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218822976Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218833508Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218844022Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=settings t=2024-03-26T02:11:13.218854522Z level=info msg="App mode production" 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=sqlstore t=2024-03-26T02:11:13.218906886Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=sqlstore t=2024-03-26T02:11:13.218930254Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.219750243Z level=info msg="Starting DB migrations" 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.219836468Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.220359841Z level=info msg="Executing migration" id="create user table" 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.220835996Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.221301819Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.221881806Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T02:11:13.311 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.222312346Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T02:11:13.312 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.222669388Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T02:11:13.312 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.223362025Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T02:11:13.312 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.223767984Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.224162323Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.224562715Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.224913951Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.225235634Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.225644349Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.225942555Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.226329912Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.226653671Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.227067305Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.227448973Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.227870421Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.228997937Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.229480963Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.230457441Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T02:11:13.313 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.23116087Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.231561007Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.232441736Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.233460571Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.234350606Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.235086768Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.236104725Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.236790642Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.23770867Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.238495916Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.239457604Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.240230915Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.24115202Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.241707125Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.24205821Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T02:11:13.314 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.242882177Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.243258062Z level=info msg="Executing migration" id="create star table" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.243622631Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.24405063Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.246530705Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.249323621Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.249705162Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.252829468Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.255567336Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.258294088Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.258752024Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.261738639Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.26453973Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.267417106Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.270258249Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T02:11:13.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.273172308Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.273731453Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.276704584Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.279392813Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.282417481Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.282855Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.285760327Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.288621925Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.291285288Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.291820493Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.294595232Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.297315216Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T02:11:13.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.299822551Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T02:11:13.317 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.30049135Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T02:11:13.317 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.303270594Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T02:11:13.317 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.305939538Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T02:11:13.317 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.308714454Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T02:11:13.317 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.309141702Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T02:11:13.562 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.311831083Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T02:11:13.562 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.312168669Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T02:11:13.562 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.314520449Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T02:11:13.562 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.317069237Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T02:11:13.562 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.317630587Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T02:11:13.562 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.318458929Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.319423407Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.320211561Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.320654359Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.321652489Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.322338382Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.323176292Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.324189565Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.32512169Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.325894664Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.326819603Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.327500004Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.328315741Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.32910062Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.330031784Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.330481184Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T02:11:13.563 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.331365865Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.332336645Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.333253185Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.333869246Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.334839026Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.335584674Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.336549872Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.337083267Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.338147397Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.339013216Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.339858303Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.340288877Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.341133085Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.341697946Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T02:11:13.564 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.342269434Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T02:11:13.565 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.343160525Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T02:11:13.565 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.34364402Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T02:11:13.566 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.344190412Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T02:11:13.566 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.345037013Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T02:11:13.566 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.345385745Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T02:11:13.566 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.348062316Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T02:11:13.566 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.351002528Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T02:11:13.566 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.351509603Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T02:11:13.566 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.35441606Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T02:11:13.566 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.357311425Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.360118629Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.360708743Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.363633432Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.36643848Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.369307619Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.372169341Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.374901271Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.377755473Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.380595929Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.383336204Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.386328991Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.389059029Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T02:11:13.567 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.391732832Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.39450209Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.397229037Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.399940925Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.4027426Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.405448704Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.407966445Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.410738591Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.413717711Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.416277358Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.41923128Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.421764886Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.424559426Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.427265846Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T02:11:13.568 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.43005697Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.432526068Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.435273981Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.436155237Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.438844212Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.441677615Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.441982074Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.44425047Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.446629779Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.449272051Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.449582437Z level=info msg="Executing migration" id="create session table" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.451791132Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.454627696Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.455168914Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.455563401Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T02:11:13.569 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.458337857Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.460782365Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.461090731Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.461369749Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.461659221Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.462081057Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.46235527Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.464851886Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.467298355Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.467924003Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.470750542Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.473174967Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.473633048Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.476060744Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T02:11:13.570 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.476897115Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.477733124Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.478699516Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.479285788Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.47967033Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.480101491Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.480471916Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.481879349Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.482299974Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.482726057Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.483055741Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.483390888Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.483791492Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.484439641Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.485110813Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T02:11:13.571 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.485781889Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.486429285Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.48686589Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.48716206Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.487434967Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.487841962Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.488277476Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.488644903Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.489042645Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.48945945Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.490132851Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.490803485Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.491128735Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.49154968Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.491962468Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.492569255Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T02:11:13.572 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.492886832Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.493291163Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.493719409Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.494042502Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.494471702Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.495273385Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.495686979Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.496302037Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.496751008Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.497599198Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.498496085Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.499197946Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.500008583Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.500865018Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.501863605Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.502593217Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T02:11:13.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.503420533Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.504980512Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.505970371Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.506700057Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.507518748Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.508384787Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.509277884Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.510266577Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.51119251Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.512046242Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.512895983Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.513833123Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.514647974Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.515486052Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.516354554Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T02:11:13.574 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.517211439Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.518066414Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.518940724Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.519926783Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.520683167Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.521643193Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.522305928Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.52323852Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.523719957Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.52412619Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.525187006Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.527870354Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.530734281Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.533689912Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.536392293Z level=info msg="Executing migration" id="create team table" 2024-03-26T02:11:13.575 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.539393565Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T02:11:13.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.542209165Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T02:11:13.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.544924535Z level=info msg="Executing migration" id="create team member table" 2024-03-26T02:11:13.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.547710015Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T02:11:13.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.550467295Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T02:11:13.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.553128239Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T02:11:13.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.55593803Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T02:11:13.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.558636065Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T02:11:13.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.56169234Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T02:11:13.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.564360841Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T02:11:13.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.567135274Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T02:11:13.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.570115234Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T02:11:13.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.572789823Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T02:11:13.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.573219547Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T02:11:13.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.576164355Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T02:11:13.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.578682422Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T02:11:13.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.581277493Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T02:11:13.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.583847696Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T02:11:13.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.586637964Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T02:11:13.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.589562349Z level=info msg="Executing migration" id="create tag table" 2024-03-26T02:11:13.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.592240775Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T02:11:13.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.595023309Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T02:11:13.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.595570494Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T02:11:13.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.598099167Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T02:11:13.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.6009922Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T02:11:13.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.603701247Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T02:11:13.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.604120603Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T02:11:13.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.60685579Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T02:11:13.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.609426488Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T02:11:13.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.612121641Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T02:11:13.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.612539711Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T02:11:13.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.61533482Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T02:11:13.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.616114224Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.61694387Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.617709699Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.620359162Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.623155791Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.623887107Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.624668887Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.625590074Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.626325933Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.627193294Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.627641871Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.628478854Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.629324581Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.630170481Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.630595992Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T02:11:13.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.631592495Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.632324361Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.633112941Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.633435589Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.634279323Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.635128439Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.636146798Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.636702263Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.637620238Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.638591605Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.639453978Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.640048052Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.641091238Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.642070417Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T02:11:13.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.642919428Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.643260148Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.644202404Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.644827957Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.645286016Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.646079483Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.64658977Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.647047286Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.647834693Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.648338073Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.64917091Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.652011077Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.652455949Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T02:11:13.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.655858906Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.659251107Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.662289381Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.662761231Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.665632013Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.668465024Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.671418588Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.671882881Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.674763195Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.677672543Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.68053055Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.681392498Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.684256814Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.687101641Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.68993052Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T02:11:13.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.690796173Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.693680862Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.69657739Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.699209273Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.700091422Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.703004005Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.705598941Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.708394457Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.709009724Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.711865817Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.714620588Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.717371488Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.718278319Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.72108866Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.723843328Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T02:11:13.820 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.726410795Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.726844034Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.729735048Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.730420731Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.731330084Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.731792376Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.732656492Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.733527322Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.734344381Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.735246643Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.736040608Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.736886078Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.737720628Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.738185349Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T02:11:13.821 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.73900147Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.73943771Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.739875992Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.739967942Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.740688428Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.741110741Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.742049464Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.742333506Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.742450234Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.74287311Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.743325142Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.744156121Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.744575083Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.745448539Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T02:11:13.822 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.746296249Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.74716715Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.74761404Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.74849071Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.749345416Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.753989659Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.755039718Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.755886648Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.760870008Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.766202826Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.767005424Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.767970372Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.768764556Z level=info msg="Executing migration" id="create permission table" 2024-03-26T02:11:13.823 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.771730784Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.774599571Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.777517776Z level=info msg="Executing migration" id="create role table" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.780582012Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.783390148Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.78671453Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.789004809Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.791697666Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.794717878Z level=info msg="Executing migration" id="create team role table" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.79749732Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.800356206Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.803286402Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.80379528Z level=info msg="Executing migration" id="create user role table" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.806602775Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.809508461Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T02:11:13.824 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.812246095Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T02:11:13.825 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.81285614Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T02:11:13.825 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.815601898Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T02:11:14.075 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.81849038Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T02:11:14.075 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.821458529Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T02:11:14.075 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.822556109Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T02:11:14.075 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.825506148Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.82815439Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.830850814Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.831308602Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.831715772Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.832187517Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.833186912Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.833637337Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.834110586Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.834406718Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.834869665Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.835349748Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.835758267Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.8361036Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T02:11:14.076 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.836461888Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.836896728Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.837363705Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.838379068Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.838716454Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.839165759Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.83963272Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.840098323Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.841165538Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.841626841Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.842077595Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.842180095Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.842521299Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.842625241Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.842934611Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T02:11:14.077 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.843372358Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T02:11:14.078 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.843797435Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:11:14.078 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.844345616Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:11:14.078 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.844813465Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T02:11:14.078 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.845249705Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.845677957Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.846430989Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.846890572Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.847351606Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.847898997Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.848354532Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.851354435Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.85237966Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.853415777Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.853778119Z level=info msg="Executing migration" id="add share column" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.854803645Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T02:11:14.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.855265932Z level=info msg="Executing migration" id="create file table" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.855720117Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.856173865Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.856668279Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.857544496Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.858529614Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.859278233Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.860036725Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.860365192Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.860905449Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.861963473Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.862823306Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.863324664Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.864193657Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.865074454Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T02:11:14.080 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.865890841Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.866974356Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.868027847Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.868905409Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.878908717Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.87939225Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.882299385Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.885839246Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.888596063Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.891432328Z level=info msg="Executing migration" id="create folder table" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.894456434Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.897105032Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.89992468Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.900246975Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T02:11:14.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=migrator t=2024-03-26T02:11:13.90307619Z level=info msg="migrations completed" performed=485 skipped=0 duration=683.262488ms 2024-03-26T02:11:14.082 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=sqlstore t=2024-03-26T02:11:13.903482916Z level=info msg="Created default organization" 2024-03-26T02:11:14.082 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=plugin.loader t=2024-03-26T02:11:13.963574503Z level=info msg="Plugin registered" pluginID=input 2024-03-26T02:11:14.082 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=plugin.loader t=2024-03-26T02:11:13.99123675Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-26T02:11:14.082 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=plugin.loader t=2024-03-26T02:11:13.991273551Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-26T02:11:14.082 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=secrets t=2024-03-26T02:11:13.991340831Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T02:11:14.082 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=query_data t=2024-03-26T02:11:13.993479114Z level=info msg="Query Service initialization" 2024-03-26T02:11:14.082 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:13 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=live.push_http t=2024-03-26T02:11:13.995826172Z level=info msg="Live Push Gateway initialization" 2024-03-26T02:11:14.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:14 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:14 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:14 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:14 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:14 smithi052 ceph-mon[25889]: Deploying daemon node-exporter.a on smithi052 2024-03-26T02:11:14.342 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:14 smithi052 ceph-mon[25889]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:14.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-mon[27627]: Deploying daemon node-exporter.a on smithi052 2024-03-26T02:11:14.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-mon[27627]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:14 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:14 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:14 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:14 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:14 smithi191 ceph-mon[27701]: Deploying daemon node-exporter.a on smithi052 2024-03-26T02:11:14.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:14 smithi191 ceph-mon[27701]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:14.743 INFO:teuthology.orchestra.run.smithi052.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:11:14.743 INFO:teuthology.orchestra.run.smithi052.stdout:agent.smithi052 smithi052 running 8s ago 3m - - 2024-03-26T02:11:14.743 INFO:teuthology.orchestra.run.smithi052.stdout:agent.smithi165 smithi165 running 12s ago 3m - - 2024-03-26T02:11:14.743 INFO:teuthology.orchestra.run.smithi052.stdout:agent.smithi191 smithi191 running 11s ago 3m - - 2024-03-26T02:11:14.743 INFO:teuthology.orchestra.run.smithi052.stdout:alertmanager.a smithi052 *:9093,9094 running (17s) 8s ago 16s 15.6M - 0.25.0 c8568f914cd2 55187f8e13ad 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:grafana.a smithi165 *:3000 starting - - - - 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.x smithi165 *:8443,8765 running (2m) 12s ago 2m 449M - 19.0.0-1775-g5e70124b 5f21c9264136 8af85cf1aead 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.y smithi052 *:9283,8765 running (4m) 8s ago 4m 500M - 19.0.0-1775-g5e70124b 5f21c9264136 280496602fcc 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.z smithi191 *:8443,8765 running (2m) 11s ago 2m 449M - 19.0.0-1775-g5e70124b 5f21c9264136 1267993bed80 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:mon.a smithi052 running (4m) 8s ago 4m 48.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 bd4ba28f3865 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:mon.b smithi165 running (3m) 12s ago 3m 36.4M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 9eb5c805c486 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:mon.c smithi191 running (3m) 11s ago 3m 34.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 3c438a495cec 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 smithi052 running (2m) 8s ago 2m 53.0M 4096M 19.0.0-1775-g5e70124b 5f21c9264136 6e213344e142 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 smithi165 running (118s) 12s ago 118s 53.1M 4096M 19.0.0-1775-g5e70124b 5f21c9264136 120224fa7331 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 smithi191 running (92s) 11s ago 91s 60.4M 4096M 19.0.0-1775-g5e70124b 5f21c9264136 30a8721b8c3c 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:prometheus.a smithi165 *:9095 running (52s) 12s ago 51s 27.1M - 2.43.0 a07b618ecd1d 51f40423f9a9 2024-03-26T02:11:14.744 INFO:teuthology.orchestra.run.smithi052.stdout:rgw.foo.a smithi052 *:80 running (73s) 8s ago 73s 78.7M - 19.0.0-1775-g5e70124b 5f21c9264136 89a19713c515 2024-03-26T02:11:14.952 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=infra.usagestats.collector t=2024-03-26T02:11:14.634846982Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=provisioning.datasources t=2024-03-26T02:11:14.635845398Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=provisioning.datasources t=2024-03-26T02:11:14.647782658Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=provisioning.alerting t=2024-03-26T02:11:14.659113206Z level=info msg="starting to provision alerting" 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=provisioning.alerting t=2024-03-26T02:11:14.659153785Z level=info msg="finished to provision alerting" 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=grafanaStorageLogger t=2024-03-26T02:11:14.65929112Z level=info msg="storage starting" 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=ngalert.state.manager t=2024-03-26T02:11:14.659436687Z level=info msg="Warming state cache for startup" 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=http.server t=2024-03-26T02:11:14.662489014Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=ngalert.state.manager t=2024-03-26T02:11:14.663345542Z level=info msg="State cache has been initialized" states=0 duration=3.907317ms 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=ngalert.multiorg.alertmanager t=2024-03-26T02:11:14.663407196Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=ticker t=2024-03-26T02:11:14.663461659Z level=info msg=starting first_tick=2024-03-26T02:11:20Z 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=sqlstore.transactions t=2024-03-26T02:11:14.685420046Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=sqlstore.transactions t=2024-03-26T02:11:14.696881486Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-26T02:11:14.953 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:11:14 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a[38663]: logger=sqlstore.transactions t=2024-03-26T02:11:14.710430781Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-26T02:11:15.397 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:15 smithi052 ceph-mon[25889]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:15.399 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-26T02:11:15.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:15 smithi165 ceph-mon[27627]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:15.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:15 smithi191 ceph-mon[27701]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:16.385 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:11:16.507 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:16 smithi052 ceph-mon[25889]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:16.536 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:16 smithi191 ceph-mon[27701]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:16.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:16 smithi165 ceph-mon[27627]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:17.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:17 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:17.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:17 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:17.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:17 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:17.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:17 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ::ffff:172.21.15.165 - - [26/Mar/2024:02:11:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:11:17.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:17 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:17.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:17 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:17.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:17 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:17 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:17 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:17.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:17 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:18.108 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:17 smithi052 bash[51330]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T02:11:18.108 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 bash[51330]: Writing manifest to image destination 2024-03-26T02:11:18.697 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 podman[51330]: 2024-03-26 02:11:18.397235836 +0000 UTC m=+3.018237704 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T02:11:18.698 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 podman[51330]: 2024-03-26 02:11:18.495504349 +0000 UTC m=+3.116506216 container create 68e297847e1bf026015a77da2f9806c277815e1d6908f4871245dce00dee1a05 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:18.961 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 podman[51330]: 2024-03-26 02:11:18.697567235 +0000 UTC m=+3.318569103 container init 68e297847e1bf026015a77da2f9806c277815e1d6908f4871245dce00dee1a05 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:18.961 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 podman[51330]: 2024-03-26 02:11:18.703483457 +0000 UTC m=+3.324485328 container start 68e297847e1bf026015a77da2f9806c277815e1d6908f4871245dce00dee1a05 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:18.961 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.707Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T02:11:18.961 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.707Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.707Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.707Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T02:11:18.962 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T02:11:18.963 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T02:11:18.964 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T02:11:18.964 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T02:11:18.964 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T02:11:18.964 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T02:11:18.964 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T02:11:18.964 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T02:11:18.964 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T02:11:18.964 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T02:11:18.964 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T02:11:18.964 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.708Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.709Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T02:11:18.965 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a[51533]: ts=2024-03-26T02:11:18.709Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T02:11:18.966 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 bash[51330]: 68e297847e1bf026015a77da2f9806c277815e1d6908f4871245dce00dee1a05 2024-03-26T02:11:18.966 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:11:18 smithi052 systemd[1]: Started Ceph node-exporter.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:11:19.305 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:19 smithi052 ceph-mon[25889]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:19.305 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:19 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.305 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:19 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.305 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:19 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.306 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:19 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.306 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:19 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.306 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:19 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:19 smithi165 ceph-mon[27627]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:19 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:19 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:19 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:19 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:19 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:19 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:19 smithi191 ceph-mon[27701]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:19 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:19 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:19 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:19 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:19 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:19.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:19 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:20.046 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:19 smithi165 systemd[1]: Starting Ceph node-exporter.b for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:20.074 INFO:teuthology.orchestra.run.smithi052.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-26T02:11:20.074 INFO:teuthology.orchestra.run.smithi052.stdout:agent 3/3 16s ago 48s * 2024-03-26T02:11:20.074 INFO:teuthology.orchestra.run.smithi052.stdout:alertmanager ?:9093,9094 1/1 13s ago 73s smithi052=a;count:1 2024-03-26T02:11:20.074 INFO:teuthology.orchestra.run.smithi052.stdout:grafana ?:3000 1/1 1s ago 70s smithi165=a;count:1 2024-03-26T02:11:20.074 INFO:teuthology.orchestra.run.smithi052.stdout:mgr 3/3 16s ago 2m smithi052=y;smithi165=x;smithi191=z;count:3 2024-03-26T02:11:20.074 INFO:teuthology.orchestra.run.smithi052.stdout:mon 3/3 16s ago 3m smithi052:172.21.15.52=a;smithi165:172.21.15.165=b;smithi191:172.21.15.191=c;count:3 2024-03-26T02:11:20.074 INFO:teuthology.orchestra.run.smithi052.stdout:node-exporter ?:9100 0/3 - 76s smithi052=a;smithi165=b;smithi191=c;count:3 2024-03-26T02:11:20.074 INFO:teuthology.orchestra.run.smithi052.stdout:osd 3 16s ago - 2024-03-26T02:11:20.074 INFO:teuthology.orchestra.run.smithi052.stdout:prometheus ?:9095 1/1 1s ago 78s smithi165=a;count:1 2024-03-26T02:11:20.074 INFO:teuthology.orchestra.run.smithi052.stdout:rgw.foo.a ?:80 1/1 13s ago 78s smithi052=foo.a;count:1 2024-03-26T02:11:20.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:20 smithi191 ceph-mon[27701]: Deploying daemon node-exporter.b on smithi165 2024-03-26T02:11:20.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:20 smithi165 ceph-mon[27627]: Deploying daemon node-exporter.b on smithi165 2024-03-26T02:11:20.452 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:20 smithi165 bash[39565]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T02:11:20.461 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:20 smithi052 ceph-mon[25889]: Deploying daemon node-exporter.b on smithi165 2024-03-26T02:11:20.886 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-26T02:11:20.952 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:20 smithi165 bash[39565]: Getting image source signatures 2024-03-26T02:11:20.952 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:20 smithi165 bash[39565]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T02:11:20.952 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:20 smithi165 bash[39565]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T02:11:20.952 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:20 smithi165 bash[39565]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T02:11:21.197 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:21 smithi052 ceph-mon[25889]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:21.197 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:21 smithi052 ceph-mon[25889]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:21.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:21 smithi165 ceph-mon[27627]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:21.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:21 smithi165 ceph-mon[27627]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:21 smithi191 ceph-mon[27701]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:21.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:21 smithi191 ceph-mon[27701]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:21.674 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:11:21.952 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:21 smithi165 bash[39565]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T02:11:21.952 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:21 smithi165 bash[39565]: Writing manifest to image destination 2024-03-26T02:11:22.394 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 podman[39565]: 2024-03-26 02:11:22.077190477 +0000 UTC m=+2.120394136 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T02:11:22.394 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 podman[39565]: 2024-03-26 02:11:22.17528132 +0000 UTC m=+2.218485024 container create 5da4ede6e78ebe84ead453a44d7daf400eb7aef0d8cb81ca92a62537d0f5bc25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:11:22.394 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 podman[39565]: 2024-03-26 02:11:22.393936352 +0000 UTC m=+2.437140010 container init 5da4ede6e78ebe84ead453a44d7daf400eb7aef0d8cb81ca92a62537d0f5bc25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:11:22.703 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 podman[39565]: 2024-03-26 02:11:22.399364324 +0000 UTC m=+2.442567983 container start 5da4ede6e78ebe84ead453a44d7daf400eb7aef0d8cb81ca92a62537d0f5bc25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:11:22.703 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.403Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T02:11:22.703 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.403Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T02:11:22.703 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T02:11:22.703 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T02:11:22.703 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T02:11:22.703 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T02:11:22.703 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T02:11:22.703 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T02:11:22.703 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.404Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T02:11:22.704 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T02:11:22.705 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T02:11:22.706 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T02:11:22.706 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T02:11:22.706 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T02:11:22.706 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T02:11:22.706 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T02:11:22.706 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T02:11:22.706 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T02:11:22.706 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T02:11:22.706 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T02:11:22.707 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T02:11:22.707 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T02:11:22.707 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T02:11:22.707 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T02:11:22.707 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T02:11:22.707 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b[39615]: ts=2024-03-26T02:11:22.405Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T02:11:22.707 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 bash[39565]: 5da4ede6e78ebe84ead453a44d7daf400eb7aef0d8cb81ca92a62537d0f5bc25 2024-03-26T02:11:22.707 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:11:22 smithi165 systemd[1]: Started Ceph node-exporter.b for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:11:23.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:23 smithi191 ceph-mon[27701]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:23.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:23 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:23.260 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:23 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:23.261 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:23 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:23.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:23 smithi165 ceph-mon[27627]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:23.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:23 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:23.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:23 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:23.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:23 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:23.453 INFO:teuthology.orchestra.run.smithi052.stdout:HOST ADDR LABELS STATUS 2024-03-26T02:11:23.453 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 172.21.15.52 2024-03-26T02:11:23.453 INFO:teuthology.orchestra.run.smithi052.stdout:smithi165 172.21.15.165 2024-03-26T02:11:23.453 INFO:teuthology.orchestra.run.smithi052.stdout:smithi191 172.21.15.191 2024-03-26T02:11:23.453 INFO:teuthology.orchestra.run.smithi052.stdout:3 hosts in cluster 2024-03-26T02:11:23.453 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:23 smithi052 ceph-mon[25889]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:23.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:23 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:23.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:23 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:23.454 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:23 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:24.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:24 smithi052 ceph-mon[25889]: Deploying daemon node-exporter.c on smithi191 2024-03-26T02:11:24.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:24 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:24.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:24 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:24.185 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:24 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:24.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:24 smithi165 ceph-mon[27627]: Deploying daemon node-exporter.c on smithi191 2024-03-26T02:11:24.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:24 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:24.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:24 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:24.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:24 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:24.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:24 smithi191 ceph-mon[27701]: Deploying daemon node-exporter.c on smithi191 2024-03-26T02:11:24.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:24 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:24.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:24 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:24.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:24 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:24.920 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-26T02:11:25.116 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:25 smithi191 ceph-mon[27701]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:25.116 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:25 smithi191 ceph-mon[27701]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:25.199 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:11:25.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:25 smithi052 ceph-mon[25889]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:25.210 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:25 smithi052 ceph-mon[25889]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:25.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:25 smithi165 ceph-mon[27627]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:25.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:25 smithi165 ceph-mon[27627]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:25.788 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:25 smithi191 bash[40311]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T02:11:26.400 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:26 smithi191 bash[40311]: Getting image source signatures 2024-03-26T02:11:26.400 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:26 smithi191 bash[40311]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T02:11:26.400 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:26 smithi191 bash[40311]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T02:11:26.400 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:26 smithi191 bash[40311]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T02:11:27.086 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:27 smithi052 ceph-mon[25889]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:27.086 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:27 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:27.086 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:27 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:27.087 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:27 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:27.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:27 smithi165 ceph-mon[27627]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:27.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:27 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:27.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:27 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:27.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:27 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:27 smithi191 ceph-mon[27701]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:27 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:27 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:27.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:27 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:27.710 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:27 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ::ffff:172.21.15.165 - - [26/Mar/2024:02:11:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:11:27.878 INFO:teuthology.orchestra.run.smithi052.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513010N480DGN 447G No 1s ago Has a FileSystem, LVM detected 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme1n1 ssd Linux_f7f1de61938e60e35fc2 89.4G Yes 1s ago 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme2n1 ssd Linux_e35b0736fab73bc61901 89.4G Yes 1s ago 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme3n1 ssd Linux_368e20502751d9c63f2c 89.4G Yes 1s ago 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme4n1 ssd Linux_dd0e295881e725b924cd 89.4G No 1s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W4HRYS 931G No 1s ago Has partitions 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi165 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CE400BGN 372G No 9s ago Has a FileSystem, LVM detected 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi165 /dev/nvme1n1 ssd Linux_f8f8be4974f01429b2fd 89.4G Yes 9s ago 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi165 /dev/nvme2n1 ssd Linux_e3edd30f285bd9641707 89.4G Yes 9s ago 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi165 /dev/nvme3n1 ssd Linux_6959243787108a82da9e 89.4G Yes 9s ago 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi165 /dev/nvme4n1 ssd Linux_7498ded24bdb4b1877e6 89.4G No 9s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi165 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XG09 931G No 9s ago Has partitions 2024-03-26T02:11:27.879 INFO:teuthology.orchestra.run.smithi052.stdout:smithi191 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303008M400BGN 372G No 4s ago Has a FileSystem, LVM detected 2024-03-26T02:11:27.880 INFO:teuthology.orchestra.run.smithi052.stdout:smithi191 /dev/nvme1n1 ssd Linux_98a21a3322fec246b29c 89.4G Yes 4s ago 2024-03-26T02:11:27.880 INFO:teuthology.orchestra.run.smithi052.stdout:smithi191 /dev/nvme2n1 ssd Linux_6ab9f5db68774c1474be 89.4G Yes 4s ago 2024-03-26T02:11:27.880 INFO:teuthology.orchestra.run.smithi052.stdout:smithi191 /dev/nvme3n1 ssd Linux_8d9fbb8547eb4597fdef 89.4G Yes 4s ago 2024-03-26T02:11:27.880 INFO:teuthology.orchestra.run.smithi052.stdout:smithi191 /dev/nvme4n1 ssd Linux_39375883a584f906e495 89.4G No 4s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-26T02:11:27.880 INFO:teuthology.orchestra.run.smithi052.stdout:smithi191 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XMDN 931G No 4s ago Has partitions 2024-03-26T02:11:28.037 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:27 smithi191 bash[40311]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T02:11:28.037 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:27 smithi191 bash[40311]: Writing manifest to image destination 2024-03-26T02:11:28.463 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 podman[40311]: 2024-03-26 02:11:28.153200209 +0000 UTC m=+2.596676884 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T02:11:28.463 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 podman[40311]: 2024-03-26 02:11:28.235624817 +0000 UTC m=+2.679101479 container create 98d57938cbc9b4a74aeee0d7f866e57168613464f0e1045f48aa10a04d4d36eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T02:11:28.463 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 podman[40311]: 2024-03-26 02:11:28.463142828 +0000 UTC m=+2.906619490 container init 98d57938cbc9b4a74aeee0d7f866e57168613464f0e1045f48aa10a04d4d36eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T02:11:28.624 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-26T02:11:28.787 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 podman[40311]: 2024-03-26 02:11:28.468703473 +0000 UTC m=+2.912180138 container start 98d57938cbc9b4a74aeee0d7f866e57168613464f0e1045f48aa10a04d4d36eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T02:11:28.787 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.472Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T02:11:28.788 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.472Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T02:11:28.788 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.473Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T02:11:28.788 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.473Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T02:11:28.788 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.473Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T02:11:28.788 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.473Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T02:11:28.788 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T02:11:28.788 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T02:11:28.788 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T02:11:28.788 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T02:11:28.788 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T02:11:28.789 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T02:11:28.789 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T02:11:28.789 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T02:11:28.789 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T02:11:28.789 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T02:11:28.789 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T02:11:28.789 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T02:11:28.789 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T02:11:28.789 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T02:11:28.789 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T02:11:28.789 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T02:11:28.790 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T02:11:28.791 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T02:11:28.791 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T02:11:28.791 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T02:11:28.791 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T02:11:28.791 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T02:11:28.791 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T02:11:28.791 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T02:11:28.791 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c[40371]: ts=2024-03-26T02:11:28.474Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T02:11:28.792 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 bash[40311]: 98d57938cbc9b4a74aeee0d7f866e57168613464f0e1045f48aa10a04d4d36eb 2024-03-26T02:11:28.793 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:11:28 smithi191 systemd[1]: Started Ceph node-exporter.c for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:11:28.887 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:11:29.273 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:29.273 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:29.273 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:11:29.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:11:29.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:11:29.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:29 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:11:29.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:29 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.537 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:29.538 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:11:29.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:11:29.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:11:29.539 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:29 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:30.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:30 smithi052 ceph-mon[25889]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:11:30.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:30 smithi052 ceph-mon[25889]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:11:30.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:30 smithi052 ceph-mon[25889]: Reconfiguring daemon alertmanager.a on smithi052 2024-03-26T02:11:30.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:30 smithi165 ceph-mon[27627]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:11:30.358 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:30 smithi165 ceph-mon[27627]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:11:30.358 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:30 smithi165 ceph-mon[27627]: Reconfiguring daemon alertmanager.a on smithi052 2024-03-26T02:11:30.421 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:30 smithi191 ceph-mon[27701]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-26T02:11:30.421 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:30 smithi191 ceph-mon[27701]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:11:30.421 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:30 smithi191 ceph-mon[27701]: Reconfiguring daemon alertmanager.a on smithi052 2024-03-26T02:11:31.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:31 smithi165 ceph-mon[27627]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:31.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:31 smithi191 ceph-mon[27701]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:31.460 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:31 smithi052 ceph-mon[25889]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:32.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:32 smithi052 ceph-mon[25889]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:32.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:32.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:32.579 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:32.580 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:32.580 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:32 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:32.580 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: agent 2024-03-26T02:11:32.580 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: agent 2024-03-26T02:11:32.580 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-26T02:11:32.580 INFO:teuthology.orchestra.run.smithi052.stdout: host_pattern: '*' 2024-03-26T02:11:32.580 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-26T02:11:32.580 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-26T02:10:31.399696Z' 2024-03-26T02:11:32.580 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-26T02:11:18.421548Z' 2024-03-26T02:11:32.580 INFO:teuthology.orchestra.run.smithi052.stdout: running: 3 2024-03-26T02:11:32.580 INFO:teuthology.orchestra.run.smithi052.stdout: size: 3 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout:events: 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout:- 2024-03-26T02:10:32.880145Z service:agent [INFO] "service was created" 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: alertmanager 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: alertmanager 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout: count: 1 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi052=a 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-26T02:10:06.552854Z' 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-26T02:11:32.093326Z' 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout: ports: 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout: - 9093 2024-03-26T02:11:32.581 INFO:teuthology.orchestra.run.smithi052.stdout: - 9094 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout: running: 1 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout: size: 1 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout:events: 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout:- 2024-03-26T02:10:58.634955Z service:alertmanager [INFO] "service was created" 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: grafana 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: grafana 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout: count: 1 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi165=a 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout:spec: 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout: anonymous_access: true 2024-03-26T02:11:32.582 INFO:teuthology.orchestra.run.smithi052.stdout: protocol: https 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-26T02:10:09.278639Z' 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-26T02:11:18.422515Z' 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout: ports: 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout: - 3000 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout: running: 1 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout: size: 1 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout:events: 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout:- 2024-03-26T02:11:13.266953Z service:grafana [INFO] "service was created" 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: mgr 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: mgr 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout: count: 3 2024-03-26T02:11:32.583 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi052=y 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi165=x 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi191=z 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-26T02:08:25.401165Z' 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-26T02:11:18.421979Z' 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout: running: 3 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout: size: 3 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: mon 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: mon 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout: count: 3 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-26T02:11:32.584 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi052:172.21.15.52=a 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi165:172.21.15.165=b 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi191:172.21.15.191=c 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-26T02:07:57.751896Z' 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-26T02:11:18.421763Z' 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout: running: 3 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout: size: 3 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: node-exporter 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: node-exporter 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-26T02:11:32.585 INFO:teuthology.orchestra.run.smithi052.stdout: count: 3 2024-03-26T02:11:32.586 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-26T02:11:32.586 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi052=a 2024-03-26T02:11:32.586 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi165=b 2024-03-26T02:11:32.586 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi191=c 2024-03-26T02:11:32.586 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-26T02:11:32.586 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-26T02:10:03.805582Z' 2024-03-26T02:11:32.586 INFO:teuthology.orchestra.run.smithi052.stdout: ports: 2024-03-26T02:11:32.586 INFO:teuthology.orchestra.run.smithi052.stdout: - 9100 2024-03-26T02:11:32.586 INFO:teuthology.orchestra.run.smithi052.stdout: running: 1 2024-03-26T02:11:32.586 INFO:teuthology.orchestra.run.smithi052.stdout: size: 3 2024-03-26T02:11:32.586 INFO:teuthology.orchestra.run.smithi052.stdout:events: 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout:- 2024-03-26T02:11:28.657304Z service:node-exporter [INFO] "service was created" 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: osd 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: osd 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout:unmanaged: true 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout:spec: 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout: filter_logic: AND 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout: objectstore: bluestore 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout: container_image_id: 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-26T02:11:18.422167Z' 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout: running: 3 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout: size: 3 2024-03-26T02:11:32.587 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: prometheus 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: prometheus 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout: count: 1 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi165=a 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-26T02:10:01.131034Z' 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-26T02:11:18.422349Z' 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout: ports: 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout: - 9095 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout: running: 1 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout: size: 1 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout:events: 2024-03-26T02:11:32.588 INFO:teuthology.orchestra.run.smithi052.stdout:- 2024-03-26T02:11:28.667188Z service:prometheus [INFO] "service was created" 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: rgw 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout:service_id: foo.a 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: rgw.foo.a 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout: count: 1 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi052=foo.a 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-26T02:10:01.549459Z' 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-26T02:11:32.093249Z' 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout: ports: 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout: - 80 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout: running: 1 2024-03-26T02:11:32.589 INFO:teuthology.orchestra.run.smithi052.stdout: size: 1 2024-03-26T02:11:32.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:32 smithi165 ceph-mon[27627]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:32.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:32.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:32.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:32.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:32.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:32 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:32 smithi191 ceph-mon[27701]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:32.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:32.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:32.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:32 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:33.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:33 smithi165 ceph-mon[27627]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T02:11:33.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:33 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:33.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:33 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:34.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:33 smithi052 ceph-mon[25889]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T02:11:34.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:33 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:34.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:33 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:33 smithi191 ceph-mon[27701]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-26T02:11:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:33 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:34.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:33 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:34.320 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-26T02:11:34.929 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:11:35.048 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:34 smithi052 systemd[1]: Stopping Ceph alertmanager.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:35.048 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:34 smithi052 ceph-mon[25889]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:35.048 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:34 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:35.048 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:34 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:35.048 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:34 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:35.193 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:34 smithi165 ceph-mon[27627]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:35.194 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:34 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:35.194 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:34 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:35.194 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:34 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:34 smithi191 ceph-mon[27701]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:34 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:34 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:35.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:34 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:35.401 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:35 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[49420]: ts=2024-03-26T02:11:35.296Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:11:35.703 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:35 smithi052 podman[53367]: 2024-03-26 02:11:35.401067175 +0000 UTC m=+0.227112768 container died 55187f8e13ad8edf039ee97bb0e222d1ab6afe2f899f1d1f1bd48822c5e4e86d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:35.961 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:35 smithi052 podman[53367]: 2024-03-26 02:11:35.80270581 +0000 UTC m=+0.628751395 container cleanup 55187f8e13ad8edf039ee97bb0e222d1ab6afe2f899f1d1f1bd48822c5e4e86d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:35.961 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:35 smithi052 bash[53367]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T02:11:36.711 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:36 smithi052 podman[53393]: 2024-03-26 02:11:36.246520799 +0000 UTC m=+0.898955821 container remove 55187f8e13ad8edf039ee97bb0e222d1ab6afe2f899f1d1f1bd48822c5e4e86d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:36.711 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:36 smithi052 podman[53393]: 2024-03-26 02:11:36.322166571 +0000 UTC m=+0.974601593 volume remove 46ac83ab827a8cc472c181c56565f1af3a704085e507ea76c362740f655f09c7 2024-03-26T02:11:36.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:36 smithi052 ceph-mon[25889]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:36.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:36 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:36.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:36 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:36.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:36 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:36 smithi191 ceph-mon[27701]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:36 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:36 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:36.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:36 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:36.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:36 smithi165 ceph-mon[27627]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:36.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:36 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:36.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:36 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:36.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:36 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:37.043 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:36 smithi052 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T02:11:37.043 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:36 smithi052 systemd[1]: Stopped Ceph alertmanager.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:11:37.043 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:36 smithi052 systemd[1]: Starting Ceph alertmanager.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:37.395 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:37 smithi052 podman[53555]: 2024-03-26 02:11:37.077987554 +0000 UTC m=+0.019171940 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T02:11:37.396 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:37 smithi052 podman[53555]: 2024-03-26 02:11:37.395356962 +0000 UTC m=+0.336541329 volume create 2a4ca10f04f7a16995d7b3f15fefe8e560d2e4dc3073c988ba9d3988b382cb4a 2024-03-26T02:11:37.535 INFO:teuthology.orchestra.run.smithi052.stdout:osd 3 5s ago - 2024-03-26T02:11:37.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:37 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: ::ffff:172.21.15.165 - - [26/Mar/2024:02:11:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:11:37.711 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:37 smithi052 podman[53555]: 2024-03-26 02:11:37.495772167 +0000 UTC m=+0.436956533 container create e1bae57f1e2f31669d04ff3e66c96056c7d30128012ee24b45d3660a9f9fdb5b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:38.294 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 podman[53555]: 2024-03-26 02:11:38.033958142 +0000 UTC m=+0.975142509 container init e1bae57f1e2f31669d04ff3e66c96056c7d30128012ee24b45d3660a9f9fdb5b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:38.295 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 podman[53555]: 2024-03-26 02:11:38.040069876 +0000 UTC m=+0.981254248 container start e1bae57f1e2f31669d04ff3e66c96056c7d30128012ee24b45d3660a9f9fdb5b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:38.295 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[53618]: ts=2024-03-26T02:11:38.053Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T02:11:38.295 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[53618]: ts=2024-03-26T02:11:38.053Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T02:11:38.295 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[53618]: ts=2024-03-26T02:11:38.054Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.52 port=9094 2024-03-26T02:11:38.295 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[53618]: ts=2024-03-26T02:11:38.055Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T02:11:38.295 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[53618]: ts=2024-03-26T02:11:38.078Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:11:38.295 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[53618]: ts=2024-03-26T02:11:38.079Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:11:38.295 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[53618]: ts=2024-03-26T02:11:38.083Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T02:11:38.295 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[53618]: ts=2024-03-26T02:11:38.083Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T02:11:38.295 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 bash[53555]: e1bae57f1e2f31669d04ff3e66c96056c7d30128012ee24b45d3660a9f9fdb5b 2024-03-26T02:11:38.295 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:38 smithi052 systemd[1]: Started Ceph alertmanager.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:11:38.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-mon[25889]: pgmap v36: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:38.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-mon[25889]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:38.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:38.645 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:38 smithi052 ceph-mon[25889]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:38.671 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T02:11:38.685 INFO:tasks.cephadm:Teardown begin 2024-03-26T02:11:38.685 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:11:38.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:11:38.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:38 smithi165 ceph-mon[27627]: pgmap v36: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:38.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:38 smithi165 ceph-mon[27627]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:38.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:38 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:38.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:38 smithi165 ceph-mon[27627]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:38.748 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:11:38.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:38 smithi191 ceph-mon[27701]: pgmap v36: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:11:38.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:38 smithi191 ceph-mon[27701]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:11:38.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:38 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:38.779 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:38 smithi191 ceph-mon[27701]: from='mgr.14412 172.21.15.52:0/1451103953' entity='mgr.y' 2024-03-26T02:11:38.785 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T02:11:38.785 DEBUG:teuthology.orchestra.run.smithi052:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:11:38.799 DEBUG:teuthology.orchestra.run.smithi165:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:11:38.813 DEBUG:teuthology.orchestra.run.smithi191:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:11:38.843 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T02:11:38.843 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T02:11:38.843 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.a 2024-03-26T02:11:38.926 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:38 smithi052 systemd[1]: Stopping Ceph mon.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:39.657 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:39 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a[25866]: 2024-03-26T02:11:39.322+0000 7f25f59af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:11:39.657 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:39 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a[25866]: 2024-03-26T02:11:39.322+0000 7f25f59af700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T02:11:39.657 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:39 smithi052 podman[53787]: 2024-03-26 02:11:39.476222974 +0000 UTC m=+0.264693901 container died bd4ba28f38651b196a674b5ec4c2901c8459f91944ee784085c5ecd5d835221c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T02:11:39.657 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:39 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:11:39] ENGINE Bus STOPPING 2024-03-26T02:11:39.658 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:39 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:11:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:11:39.658 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:39 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:11:39] ENGINE Bus STOPPED 2024-03-26T02:11:39.658 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:39 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:11:39] ENGINE Bus STARTING 2024-03-26T02:11:39.658 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:39 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:11:39] ENGINE Serving on http://:::9283 2024-03-26T02:11:39.658 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:39 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: [26/Mar/2024:02:11:39] ENGINE Bus STARTED 2024-03-26T02:11:39.960 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:39 smithi052 podman[53787]: 2024-03-26 02:11:39.829087407 +0000 UTC m=+0.617558322 container cleanup bd4ba28f38651b196a674b5ec4c2901c8459f91944ee784085c5ecd5d835221c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T02:11:39.961 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:39 smithi052 bash[53787]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a 2024-03-26T02:11:40.398 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:40 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[53618]: ts=2024-03-26T02:11:40.056Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000299713s 2024-03-26T02:11:40.711 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 26 02:11:40 smithi052 podman[53840]: 2024-03-26 02:11:40.625328274 +0000 UTC m=+1.148108140 container remove bd4ba28f38651b196a674b5ec4c2901c8459f91944ee784085c5ecd5d835221c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-a, RELEASE=squid-5e70124, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:11:40.799 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T02:11:40.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:11:40.851 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T02:11:40.851 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-26T02:11:40.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.b 2024-03-26T02:11:41.185 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:40 smithi165 systemd[1]: Stopping Ceph mon.b for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:44.179 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:43 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-b[27605]: 2024-03-26T02:11:43.737+0000 7fd34301b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:11:44.179 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:43 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-b[27605]: 2024-03-26T02:11:43.737+0000 7fd34301b700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T02:11:44.180 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:44 smithi165 podman[40694]: 2024-03-26 02:11:44.179009314 +0000 UTC m=+0.538412059 container died 9eb5c805c486e165023bae6308f2167fa59a0d5a566d82c0f6819ac90537d13e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T02:11:44.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:44 smithi165 podman[40694]: 2024-03-26 02:11:44.58645206 +0000 UTC m=+0.945854805 container cleanup 9eb5c805c486e165023bae6308f2167fa59a0d5a566d82c0f6819ac90537d13e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-b, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T02:11:44.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:44 smithi165 bash[40694]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-b 2024-03-26T02:11:45.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Mar 26 02:11:44 smithi165 podman[40713]: 2024-03-26 02:11:44.762384281 +0000 UTC m=+0.917346610 container remove 9eb5c805c486e165023bae6308f2167fa59a0d5a566d82c0f6819ac90537d13e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T02:11:45.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T02:11:45.486 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 systemd[1]: Stopping Ceph prometheus.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:45.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:11:45.488 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-26T02:11:45.488 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-26T02:11:45.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.c 2024-03-26T02:11:45.787 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:45 smithi191 systemd[1]: Stopping Ceph mon.c for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:45.835 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.612Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:11:45.835 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.612Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:11:45.835 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.612Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:11:45.835 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.612Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:11:45.835 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.612Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:11:45.835 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.612Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T02:11:45.835 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.612Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:11:45.836 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.612Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T02:11:45.836 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.612Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T02:11:45.836 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.613Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:11:45.836 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.613Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T02:11:45.836 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[35991]: ts=2024-03-26T02:11:45.613Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T02:11:45.836 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 podman[40843]: 2024-03-26 02:11:45.623050464 +0000 UTC m=+0.115488863 container died 51f40423f9a900dcb805a09274ab29662a6545c45559ed2fb4e8ff1d2de51af0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:46.185 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 podman[40843]: 2024-03-26 02:11:45.917135119 +0000 UTC m=+0.409573517 container cleanup 51f40423f9a900dcb805a09274ab29662a6545c45559ed2fb4e8ff1d2de51af0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:46.185 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:45 smithi165 bash[40843]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:11:46.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:45 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-c[27678]: 2024-03-26T02:11:45.787+0000 7fe3bebc8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:11:46.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:45 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-c[27678]: 2024-03-26T02:11:45.787+0000 7fe3bebc8700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T02:11:46.287 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:45 smithi191 podman[41074]: 2024-03-26 02:11:45.972136096 +0000 UTC m=+0.297053040 container died 3c438a495cec85c07e9f66b706c7374222c3e288580a630c9fdc94e3dc9d45c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T02:11:46.452 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 podman[40862]: 2024-03-26 02:11:46.185298129 +0000 UTC m=+0.563662036 container remove 51f40423f9a900dcb805a09274ab29662a6545c45559ed2fb4e8ff1d2de51af0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:46.452 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T02:11:46.452 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 systemd[1]: Stopped Ceph prometheus.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:11:46.452 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.160s CPU time. 2024-03-26T02:11:46.452 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 systemd[1]: Starting Ceph prometheus.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:46.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:46 smithi191 podman[41074]: 2024-03-26 02:11:46.386693848 +0000 UTC m=+0.711610794 container cleanup 3c438a495cec85c07e9f66b706c7374222c3e288580a630c9fdc94e3dc9d45c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-c, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T02:11:46.614 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 26 02:11:46 smithi191 bash[41074]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mon-c 2024-03-26T02:11:46.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T02:11:46.801 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 podman[40962]: 2024-03-26 02:11:46.449639916 +0000 UTC m=+0.018864761 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T02:11:46.801 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 podman[40962]: 2024-03-26 02:11:46.569481706 +0000 UTC m=+0.138706570 container create f147d2157231c0f18d39f15fa2f16608e625cd5f9d2e765b703f53f8829f65da (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:46.801 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 podman[40962]: 2024-03-26 02:11:46.771356272 +0000 UTC m=+0.340581094 container init f147d2157231c0f18d39f15fa2f16608e625cd5f9d2e765b703f53f8829f65da (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:46.801 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 podman[40962]: 2024-03-26 02:11:46.776523888 +0000 UTC m=+0.345748713 container start f147d2157231c0f18d39f15fa2f16608e625cd5f9d2e765b703f53f8829f65da (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:11:46.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:11:46.805 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-26T02:11:46.805 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T02:11:46.805 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.y 2024-03-26T02:11:47.157 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:46 smithi052 systemd[1]: Stopping Ceph mgr.y for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:47.187 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.800Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T02:11:47.187 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.800Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T02:11:47.187 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.800Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi165 (none))" 2024-03-26T02:11:47.187 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.801Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:11:47.187 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.801Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:11:47.187 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.802Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:11:47.187 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.803Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T02:11:47.187 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.804Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T02:11:47.187 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.804Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T02:11:47.188 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.806Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:11:47.188 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.806Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.93µs 2024-03-26T02:11:47.188 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.806Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:11:47.188 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.806Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T02:11:47.188 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.807Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T02:11:47.188 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.807Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.936µs wal_replay_duration=796.439µs wbl_replay_duration=86ns total_replay_duration=853.625µs 2024-03-26T02:11:47.189 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.808Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:11:47.189 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.808Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T02:11:47.189 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.808Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:11:47.189 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.826Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.401283ms db_storage=670ns remote_storage=1.239µs web_handler=435ns query_engine=890ns scrape=447.527µs scrape_sd=67.274µs notify=32.099µs notify_sd=10.867µs rules=17.635286ms tracing=5.641µs 2024-03-26T02:11:47.189 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.826Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T02:11:47.189 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:11:46.826Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T02:11:47.189 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 bash[40962]: f147d2157231c0f18d39f15fa2f16608e625cd5f9d2e765b703f53f8829f65da 2024-03-26T02:11:47.189 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:11:46 smithi165 systemd[1]: Started Ceph prometheus.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:11:47.414 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:47 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y[26145]: 2024-03-26T02:11:47.155+0000 7ff488f24700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:11:47.414 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:47 smithi052 podman[54036]: 2024-03-26 02:11:47.187094795 +0000 UTC m=+0.169955796 container died 280496602fcc91698097fb9dbcbd52eb037881a123df67e9f5525ca7b16ed63e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T02:11:47.710 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:47 smithi052 podman[54036]: 2024-03-26 02:11:47.628443455 +0000 UTC m=+0.611304463 container cleanup 280496602fcc91698097fb9dbcbd52eb037881a123df67e9f5525ca7b16ed63e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124) 2024-03-26T02:11:47.711 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:47 smithi052 bash[54036]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y 2024-03-26T02:11:48.058 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T02:11:48.083 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:47 smithi052 podman[54076]: 2024-03-26 02:11:47.821570713 +0000 UTC m=+0.629498389 container remove 280496602fcc91698097fb9dbcbd52eb037881a123df67e9f5525ca7b16ed63e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T02:11:48.083 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:48 smithi052 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.y.service: Deactivated successfully. 2024-03-26T02:11:48.083 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:48 smithi052 systemd[1]: Stopped Ceph mgr.y for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:11:48.083 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 26 02:11:48 smithi052 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.y.service: Consumed 1min 21.072s CPU time. 2024-03-26T02:11:48.084 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:11:48 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[53618]: ts=2024-03-26T02:11:48.058Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002327245s 2024-03-26T02:11:48.098 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:11:48.098 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T02:11:48.098 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T02:11:48.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.x 2024-03-26T02:11:48.397 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:11:48 smithi165 systemd[1]: Stopping Ceph mgr.x for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:48.702 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:11:48 smithi165 podman[41057]: 2024-03-26 02:11:48.397200518 +0000 UTC m=+0.109827804 container died 8af85cf1aead122b797ab38638d1ab7466e5fc082b32c5736f7b8438b94066bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:11:49.113 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:11:48 smithi165 podman[41057]: 2024-03-26 02:11:48.714670094 +0000 UTC m=+0.427297382 container cleanup 8af85cf1aead122b797ab38638d1ab7466e5fc082b32c5736f7b8438b94066bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:11:49.113 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:11:48 smithi165 bash[41057]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x 2024-03-26T02:11:49.113 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:11:48 smithi165 podman[41077]: 2024-03-26 02:11:48.907629356 +0000 UTC m=+0.506176329 container remove 8af85cf1aead122b797ab38638d1ab7466e5fc082b32c5736f7b8438b94066bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-x, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T02:11:49.113 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Mar 26 02:11:48 smithi165 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:11:49.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T02:11:49.168 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:11:49.169 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T02:11:49.169 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-26T02:11:49.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.z 2024-03-26T02:11:49.481 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:11:49 smithi191 systemd[1]: Stopping Ceph mgr.z for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:49.787 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:11:49 smithi191 podman[41204]: 2024-03-26 02:11:49.481765577 +0000 UTC m=+0.127986843 container died 1267993bed80a6a8833a2fd7ff26adb694c5ec954286fe7ee064074a6859446a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T02:11:50.147 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:11:49 smithi191 podman[41204]: 2024-03-26 02:11:49.799571846 +0000 UTC m=+0.445793105 container cleanup 1267993bed80a6a8833a2fd7ff26adb694c5ec954286fe7ee064074a6859446a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T02:11:50.148 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:11:49 smithi191 bash[41204]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z 2024-03-26T02:11:50.148 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:11:49 smithi191 podman[41223]: 2024-03-26 02:11:49.959876809 +0000 UTC m=+0.474206433 container remove 1267993bed80a6a8833a2fd7ff26adb694c5ec954286fe7ee064074a6859446a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-mgr-z, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-5e70124) 2024-03-26T02:11:50.148 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 26 02:11:49 smithi191 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:11:50.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-26T02:11:50.200 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:11:50.200 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-26T02:11:50.201 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T02:11:50.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@osd.0 2024-03-26T02:11:50.513 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:50 smithi052 systemd[1]: Stopping Ceph osd.0 for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:50.961 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:50 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0[38496]: 2024-03-26T02:11:50.692+0000 7f80a85b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:11:50.961 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:50 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0[38496]: 2024-03-26T02:11:50.692+0000 7f80a85b6700 -1 osd.0 31 *** Got signal Terminated *** 2024-03-26T02:11:50.961 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:50 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0[38496]: 2024-03-26T02:11:50.692+0000 7f80a85b6700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:11:56.530 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:56 smithi052 podman[54592]: 2024-03-26 02:11:56.251041116 +0000 UTC m=+5.752795170 container died 6e213344e142ec19566e69bb20846bc8f7fdf3afd03366c0715eae10efc3fb38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:11:56.530 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:56 smithi052 podman[54592]: 2024-03-26 02:11:56.529307979 +0000 UTC m=+6.031062032 container cleanup 6e213344e142ec19566e69bb20846bc8f7fdf3afd03366c0715eae10efc3fb38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-03-26T02:11:56.960 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:56 smithi052 bash[54592]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0 2024-03-26T02:11:56.961 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:56 smithi052 podman[54831]: 2024-03-26 02:11:56.738842432 +0000 UTC m=+0.490861601 container remove 6e213344e142ec19566e69bb20846bc8f7fdf3afd03366c0715eae10efc3fb38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:11:57.383 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:57 smithi052 podman[54929]: 2024-03-26 02:11:56.97255442 +0000 UTC m=+0.017658928 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:11:57.383 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:57 smithi052 podman[54929]: 2024-03-26 02:11:57.072847587 +0000 UTC m=+0.117952080 container create bfc003be8e41d9e4ff75105762cb351a57f3b3552bdd5f56a6fb26c6e5857f8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T02:11:57.384 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:57 smithi052 podman[54929]: 2024-03-26 02:11:57.283822283 +0000 UTC m=+0.328926779 container init bfc003be8e41d9e4ff75105762cb351a57f3b3552bdd5f56a6fb26c6e5857f8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:11:57.384 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:57 smithi052 podman[54929]: 2024-03-26 02:11:57.289257419 +0000 UTC m=+0.334361912 container start bfc003be8e41d9e4ff75105762cb351a57f3b3552bdd5f56a6fb26c6e5857f8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T02:11:57.384 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:57 smithi052 podman[54929]: 2024-03-26 02:11:57.383470029 +0000 UTC m=+0.428574540 container attach bfc003be8e41d9e4ff75105762cb351a57f3b3552bdd5f56a6fb26c6e5857f8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-26T02:11:57.710 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 26 02:11:57 smithi052 podman[54929]: 2024-03-26 02:11:57.589525795 +0000 UTC m=+0.634630280 container died bfc003be8e41d9e4ff75105762cb351a57f3b3552bdd5f56a6fb26c6e5857f8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:11:58.203 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T02:11:58.244 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:11:58.244 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T02:11:58.244 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T02:11:58.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@osd.1 2024-03-26T02:11:58.553 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:11:58 smithi165 systemd[1]: Stopping Ceph osd.1 for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:11:58.952 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:11:58 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1[33193]: 2024-03-26T02:11:58.551+0000 7f5ddc979700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:11:58.952 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:11:58 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1[33193]: 2024-03-26T02:11:58.551+0000 7f5ddc979700 -1 osd.1 31 *** Got signal Terminated *** 2024-03-26T02:11:58.952 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:11:58 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1[33193]: 2024-03-26T02:11:58.551+0000 7f5ddc979700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:12:04.267 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:03 smithi165 podman[41933]: 2024-03-26 02:12:03.995967727 +0000 UTC m=+5.553577335 container died 120224fa733181eacdbf34cf523f9124440b5d1988617b52094db241fa648b3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T02:12:04.608 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:04 smithi165 podman[41933]: 2024-03-26 02:12:04.365007472 +0000 UTC m=+5.922617069 container cleanup 120224fa733181eacdbf34cf523f9124440b5d1988617b52094db241fa648b3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-03-26T02:12:04.608 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:04 smithi165 bash[41933]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1 2024-03-26T02:12:04.931 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:04 smithi165 podman[41954]: 2024-03-26 02:12:04.607810685 +0000 UTC m=+0.611329184 container remove 120224fa733181eacdbf34cf523f9124440b5d1988617b52094db241fa648b3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:12:05.202 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:04 smithi165 podman[42049]: 2024-03-26 02:12:04.834327021 +0000 UTC m=+0.017799623 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:12:05.202 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:04 smithi165 podman[42049]: 2024-03-26 02:12:04.933501508 +0000 UTC m=+0.116974137 container create 2998ce510256ccb43bdc0788c4ab744dc5a0ec92fac85af4bb545a73dc06c86d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T02:12:05.202 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:05 smithi165 podman[42049]: 2024-03-26 02:12:05.160849414 +0000 UTC m=+0.344321999 container init 2998ce510256ccb43bdc0788c4ab744dc5a0ec92fac85af4bb545a73dc06c86d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T02:12:05.202 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:05 smithi165 podman[42049]: 2024-03-26 02:12:05.166215615 +0000 UTC m=+0.349688202 container start 2998ce510256ccb43bdc0788c4ab744dc5a0ec92fac85af4bb545a73dc06c86d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-26T02:12:05.702 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:05 smithi165 podman[42049]: 2024-03-26 02:12:05.252341146 +0000 UTC m=+0.435813737 container attach 2998ce510256ccb43bdc0788c4ab744dc5a0ec92fac85af4bb545a73dc06c86d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:12:05.702 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:05 smithi165 podman[42049]: 2024-03-26 02:12:05.468632072 +0000 UTC m=+0.652104648 container died 2998ce510256ccb43bdc0788c4ab744dc5a0ec92fac85af4bb545a73dc06c86d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T02:12:05.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T02:12:06.003 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:05 smithi165 podman[42094]: 2024-03-26 02:12:05.979840465 +0000 UTC m=+0.506310513 container remove 2998ce510256ccb43bdc0788c4ab744dc5a0ec92fac85af4bb545a73dc06c86d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:12:06.003 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:05 smithi165 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-26T02:12:06.003 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:05 smithi165 systemd[1]: Stopped Ceph osd.1 for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:12:06.004 INFO:journalctl@ceph.osd.1.smithi165.stdout:Mar 26 02:12:05 smithi165 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@osd.1.service: Consumed 4.398s CPU time. 2024-03-26T02:12:06.042 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:12:06.042 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T02:12:06.043 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T02:12:06.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@osd.2 2024-03-26T02:12:06.364 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:06 smithi191 systemd[1]: Stopping Ceph osd.2 for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:12:06.787 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:06 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2[34671]: 2024-03-26T02:12:06.362+0000 7f88a6f69700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:12:06.787 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:06 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2[34671]: 2024-03-26T02:12:06.362+0000 7f88a6f69700 -1 osd.2 31 *** Got signal Terminated *** 2024-03-26T02:12:06.787 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:06 smithi191 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2[34671]: 2024-03-26T02:12:06.362+0000 7f88a6f69700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:12:12.164 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:11 smithi191 podman[41905]: 2024-03-26 02:12:11.85251776 +0000 UTC m=+5.611406389 container died 30a8721b8c3c79a09843dabd0312df1c71c74892986068c009d19801f09abd5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124) 2024-03-26T02:12:12.537 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:12 smithi191 podman[41905]: 2024-03-26 02:12:12.236747452 +0000 UTC m=+5.995636063 container cleanup 30a8721b8c3c79a09843dabd0312df1c71c74892986068c009d19801f09abd5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-03-26T02:12:12.537 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:12 smithi191 bash[41905]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2 2024-03-26T02:12:13.038 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:12 smithi191 podman[41941]: 2024-03-26 02:12:12.57991304 +0000 UTC m=+0.826604834 container remove 30a8721b8c3c79a09843dabd0312df1c71c74892986068c009d19801f09abd5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:12:13.038 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:12 smithi191 bash[42008]: Error: no container with name or ID "ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2" found: no such container 2024-03-26T02:12:13.705 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:13 smithi191 podman[42109]: 2024-03-26 02:12:13.484688694 +0000 UTC m=+0.018653043 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:12:13.973 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:13 smithi191 podman[42109]: 2024-03-26 02:12:13.812363191 +0000 UTC m=+0.346327539 container create 8bd6e324db083cbb228b0c64cd2e8e8ac236c7c06f349b2671a67abf41dfd0e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-26T02:12:14.240 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:14 smithi191 podman[42109]: 2024-03-26 02:12:14.048478102 +0000 UTC m=+0.582442451 container init 8bd6e324db083cbb228b0c64cd2e8e8ac236c7c06f349b2671a67abf41dfd0e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T02:12:14.240 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:14 smithi191 podman[42109]: 2024-03-26 02:12:14.054116132 +0000 UTC m=+0.588080483 container start 8bd6e324db083cbb228b0c64cd2e8e8ac236c7c06f349b2671a67abf41dfd0e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-26T02:12:14.241 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:14 smithi191 podman[42109]: 2024-03-26 02:12:14.157677972 +0000 UTC m=+0.691642327 container attach 8bd6e324db083cbb228b0c64cd2e8e8ac236c7c06f349b2671a67abf41dfd0e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:12:14.537 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 26 02:12:14 smithi191 podman[42109]: 2024-03-26 02:12:14.363892707 +0000 UTC m=+0.897857065 container died 8bd6e324db083cbb228b0c64cd2e8e8ac236c7c06f349b2671a67abf41dfd0e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-26T02:12:15.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T02:12:15.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:12:15.111 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T02:12:15.111 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T02:12:15.112 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T02:12:15.414 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:12:15 smithi052 systemd[1]: Stopping Ceph rgw.foo.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:12:15.710 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:12:15 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-rgw-foo-a[41753]: 2024-03-26T02:12:15.412+0000 7f8f0fe3a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:12:15.711 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:12:15 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-rgw-foo-a[41753]: 2024-03-26T02:12:15.412+0000 7f8f1e6b7ac0 -1 shutting down 2024-03-26T02:12:25.711 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:12:25 smithi052 bash[55753]: time="2024-03-26T02:12:25Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T02:12:25.711 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:12:25 smithi052 podman[55753]: 2024-03-26 02:12:25.440874783 +0000 UTC m=+10.133240676 container died 89a19713c515a773a968546d9a57cfe17a61871c40b3afbfc3f0967c3d78f8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T02:12:26.058 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:12:25 smithi052 podman[55753]: 2024-03-26 02:12:25.918170647 +0000 UTC m=+10.610536538 container cleanup 89a19713c515a773a968546d9a57cfe17a61871c40b3afbfc3f0967c3d78f8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-rgw-foo-a, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T02:12:26.058 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:12:25 smithi052 bash[55753]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T02:12:26.387 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:12:26 smithi052 podman[55778]: 2024-03-26 02:12:26.169578411 +0000 UTC m=+0.722766109 container remove 89a19713c515a773a968546d9a57cfe17a61871c40b3afbfc3f0967c3d78f8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T02:12:26.388 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 26 02:12:26 smithi052 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T02:12:26.404 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T02:12:26.445 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:12:26.445 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T02:12:26.445 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T02:12:26.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T02:12:26.752 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 systemd[1]: Stopping Ceph prometheus.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:12:27.133 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.752Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.752Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.752Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.752Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.752Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.752Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.752Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.752Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.753Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.753Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.753Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a[40976]: ts=2024-03-26T02:12:26.753Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T02:12:27.134 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:26 smithi165 podman[42857]: 2024-03-26 02:12:26.763119126 +0000 UTC m=+0.119723815 container died f147d2157231c0f18d39f15fa2f16608e625cd5f9d2e765b703f53f8829f65da (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:12:27.452 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:27 smithi165 podman[42857]: 2024-03-26 02:12:27.2240862 +0000 UTC m=+0.580690896 container cleanup f147d2157231c0f18d39f15fa2f16608e625cd5f9d2e765b703f53f8829f65da (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:12:27.452 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Mar 26 02:12:27 smithi165 bash[42857]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:12:27.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T02:12:27.679 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:12:27.680 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T02:12:27.680 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T02:12:28.101 INFO:teuthology.orchestra.run.smithi052.stdout:Deleting cluster with fsid: 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:12:29.940 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:12:29 smithi052 systemd[1]: Stopping Ceph alertmanager.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:12:30.211 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:12:29 smithi052 ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a[53618]: ts=2024-03-26T02:12:29.939Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:12:30.211 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:12:29 smithi052 podman[56261]: 2024-03-26 02:12:29.960615904 +0000 UTC m=+0.118034668 container died e1bae57f1e2f31669d04ff3e66c96056c7d30128012ee24b45d3660a9f9fdb5b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:12:30.563 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:12:30 smithi052 podman[56261]: 2024-03-26 02:12:30.387519621 +0000 UTC m=+0.544938377 container cleanup e1bae57f1e2f31669d04ff3e66c96056c7d30128012ee24b45d3660a9f9fdb5b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:12:30.564 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:12:30 smithi052 bash[56261]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T02:12:30.942 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:12:30 smithi052 podman[56280]: 2024-03-26 02:12:30.563588741 +0000 UTC m=+0.601727587 container remove e1bae57f1e2f31669d04ff3e66c96056c7d30128012ee24b45d3660a9f9fdb5b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:12:30.943 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:12:30 smithi052 podman[56280]: 2024-03-26 02:12:30.664407638 +0000 UTC m=+0.702546484 volume remove 2a4ca10f04f7a16995d7b3f15fefe8e560d2e4dc3073c988ba9d3988b382cb4a 2024-03-26T02:12:31.211 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:12:31 smithi052 systemd[1]: Stopping Ceph node-exporter.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:12:31.211 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:12:30 smithi052 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T02:12:31.211 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 26 02:12:30 smithi052 systemd[1]: Stopped Ceph alertmanager.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:12:31.659 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:12:31 smithi052 podman[56398]: 2024-03-26 02:12:31.405622413 +0000 UTC m=+0.118774136 container died 68e297847e1bf026015a77da2f9806c277815e1d6908f4871245dce00dee1a05 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:12:31.943 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:12:31 smithi052 podman[56398]: 2024-03-26 02:12:31.733550607 +0000 UTC m=+0.446702320 container cleanup 68e297847e1bf026015a77da2f9806c277815e1d6908f4871245dce00dee1a05 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:12:31.943 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:12:31 smithi052 bash[56398]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T02:12:32.211 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:12:31 smithi052 podman[56417]: 2024-03-26 02:12:31.943034135 +0000 UTC m=+0.538992536 container remove 68e297847e1bf026015a77da2f9806c277815e1d6908f4871245dce00dee1a05 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:12:32.211 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:12:31 smithi052 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:12:32.211 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:12:32 smithi052 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T02:12:32.211 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:12:32 smithi052 systemd[1]: Stopped Ceph node-exporter.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:12:32.211 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 26 02:12:32 smithi052 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 1.866s CPU time. 2024-03-26T02:12:32.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T02:12:34.203 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:12:35.453 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:12:35 smithi165 systemd[1]: Stopping Ceph grafana.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:12:35.857 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:12:35 smithi165 podman[43584]: 2024-03-26 02:12:35.569291982 +0000 UTC m=+0.133438362 container died cb607c912b0472841df6abc63207ab4b1a73591a421e14452d6d8f8357a36786 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, release=239.1651231664, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, vcs-type=git, com.redhat.component=ubi8-container) 2024-03-26T02:12:36.189 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:12:35 smithi165 podman[43584]: 2024-03-26 02:12:35.938347711 +0000 UTC m=+0.502494078 container cleanup cb607c912b0472841df6abc63207ab4b1a73591a421e14452d6d8f8357a36786 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., description=Ceph Grafana Container, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, architecture=x86_64, version=8.5, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-26T02:12:36.189 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:12:35 smithi165 bash[43584]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a 2024-03-26T02:12:36.452 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:12:36 smithi165 podman[43609]: 2024-03-26 02:12:36.189165541 +0000 UTC m=+0.621182423 container remove cb607c912b0472841df6abc63207ab4b1a73591a421e14452d6d8f8357a36786 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, maintainer=Paul Cuzner , distribution-scope=public, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, description=Ceph Grafana Container) 2024-03-26T02:12:36.452 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:12:36 smithi165 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:12:36.453 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:12:36 smithi165 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T02:12:36.453 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:12:36 smithi165 systemd[1]: Stopped Ceph grafana.a for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:12:36.453 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Mar 26 02:12:36 smithi165 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@grafana.a.service: Consumed 3.837s CPU time. 2024-03-26T02:12:36.798 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:12:36 smithi165 systemd[1]: Stopping Ceph node-exporter.b for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:12:37.059 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:12:36 smithi165 podman[43729]: 2024-03-26 02:12:36.797630183 +0000 UTC m=+0.126494455 container died 5da4ede6e78ebe84ead453a44d7daf400eb7aef0d8cb81ca92a62537d0f5bc25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:12:37.337 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:12:37 smithi165 podman[43729]: 2024-03-26 02:12:37.158346782 +0000 UTC m=+0.487211061 container cleanup 5da4ede6e78ebe84ead453a44d7daf400eb7aef0d8cb81ca92a62537d0f5bc25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:12:37.337 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:12:37 smithi165 bash[43729]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T02:12:37.692 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:12:37 smithi165 podman[43747]: 2024-03-26 02:12:37.434324785 +0000 UTC m=+0.637759741 container remove 5da4ede6e78ebe84ead453a44d7daf400eb7aef0d8cb81ca92a62537d0f5bc25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:12:37.692 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:12:37 smithi165 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:12:37.692 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:12:37 smithi165 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T02:12:37.692 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:12:37 smithi165 systemd[1]: Stopped Ceph node-exporter.b for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:12:37.692 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Mar 26 02:12:37 smithi165 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 1.833s CPU time. 2024-03-26T02:12:38.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T02:12:38.395 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:12:39.537 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:12:39 smithi191 systemd[1]: Stopping Ceph node-exporter.c for 46f2b2dc-eb15-11ee-95d0-87774f69a715... 2024-03-26T02:12:39.537 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:12:39 smithi191 podman[43149]: 2024-03-26 02:12:39.48548232 +0000 UTC m=+0.098912488 container died 98d57938cbc9b4a74aeee0d7f866e57168613464f0e1045f48aa10a04d4d36eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T02:12:40.037 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:12:39 smithi191 podman[43149]: 2024-03-26 02:12:39.86364871 +0000 UTC m=+0.477078920 container cleanup 98d57938cbc9b4a74aeee0d7f866e57168613464f0e1045f48aa10a04d4d36eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T02:12:40.037 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:12:39 smithi191 bash[43149]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c 2024-03-26T02:12:40.037 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:12:40 smithi191 podman[43168]: 2024-03-26 02:12:40.014554313 +0000 UTC m=+0.533587779 container remove 98d57938cbc9b4a74aeee0d7f866e57168613464f0e1045f48aa10a04d4d36eb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-26T02:12:40.037 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:12:40 smithi191 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.c.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:12:40.537 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:12:40 smithi191 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.c.service: Failed with result 'exit-code'. 2024-03-26T02:12:40.537 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:12:40 smithi191 systemd[1]: Stopped Ceph node-exporter.c for 46f2b2dc-eb15-11ee-95d0-87774f69a715. 2024-03-26T02:12:40.537 INFO:journalctl@ceph.node-exporter.c.smithi191.stdout:Mar 26 02:12:40 smithi191 systemd[1]: ceph-46f2b2dc-eb15-11ee-95d0-87774f69a715@node-exporter.c.service: Consumed 1.849s CPU time. 2024-03-26T02:12:40.875 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:12:40.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:12:40.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:12:40.975 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T02:12:40.977 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944/remote/smithi052/crash 2024-03-26T02:12:40.978 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/crash -- . 2024-03-26T02:12:41.017 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T02:12:41.018 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now 2024-03-26T02:12:41.019 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944/remote/smithi165/crash 2024-03-26T02:12:41.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/crash -- . 2024-03-26T02:12:41.067 INFO:teuthology.orchestra.run.smithi165.stderr:tar: /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T02:12:41.067 INFO:teuthology.orchestra.run.smithi165.stderr:tar: Error is not recoverable: exiting now 2024-03-26T02:12:41.069 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944/remote/smithi191/crash 2024-03-26T02:12:41.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/crash -- . 2024-03-26T02:12:41.108 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T02:12:41.108 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now 2024-03-26T02:12:41.109 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T02:12:41.109 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:12:41.141 INFO:teuthology.orchestra.run.smithi052.stdout:2024-03-26T02:08:17.465556+0000 mon.a (mon.0) 249 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T02:12:41.142 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T02:12:41.142 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[SEC\]' /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:12:41.214 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]' /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:12:41.287 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[WRN\]' /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:12:41.357 INFO:teuthology.orchestra.run.smithi052.stdout:2024-03-26T02:08:17.465556+0000 mon.a (mon.0) 249 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T02:12:41.358 INFO:tasks.cephadm:Compressing logs... 2024-03-26T02:12:41.358 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T02:12:41.400 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T02:12:41.402 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T02:12:41.426 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T02:12:41.426 INFO:teuthology.orchestra.run.smithi052.stderr:: No such file or directory 2024-03-26T02:12:41.426 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T02:12:41.426 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T02:12:41.426 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T02:12:41.428 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mon.a.log: /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T02:12:41.428 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T02:12:41.428 INFO:teuthology.orchestra.run.smithi052.stderr: 57.8% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T02:12:41.429 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T02:12:41.429 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log 2024-03-26T02:12:41.429 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T02:12:41.429 INFO:teuthology.orchestra.run.smithi052.stderr: 89.9% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T02:12:41.429 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log: 82.2% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T02:12:41.429 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-volume.log: 85.3% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T02:12:41.430 INFO:teuthology.orchestra.run.smithi052.stderr: 84.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T02:12:41.431 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T02:12:41.431 INFO:teuthology.orchestra.run.smithi191.stderr:: No such file or directory 2024-03-26T02:12:41.431 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T02:12:41.431 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-26T02:12:41.431 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T02:12:41.435 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log 2024-03-26T02:12:41.435 INFO:teuthology.orchestra.run.smithi191.stderr:gzip/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.cephadm.log: -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T02:12:41.435 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log: 80.7% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T02:12:41.435 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T02:12:41.435 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T02:12:41.435 INFO:teuthology.orchestra.run.smithi191.stderr: 85.1% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T02:12:41.435 INFO:teuthology.orchestra.run.smithi191.stderr: 90.1% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T02:12:41.435 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mon.c.log: 76.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T02:12:41.436 INFO:teuthology.orchestra.run.smithi165.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T02:12:41.436 INFO:teuthology.orchestra.run.smithi165.stderr:: No such file or directory 2024-03-26T02:12:41.436 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T02:12:41.436 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T02:12:41.436 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T02:12:41.436 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T02:12:41.436 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log 2024-03-26T02:12:41.436 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mgr.x.log: /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T02:12:41.437 INFO:teuthology.orchestra.run.smithi165.stderr: 90.1% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T02:12:41.437 INFO:teuthology.orchestra.run.smithi165.stderr: 80.7% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T02:12:41.437 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T02:12:41.437 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-volume.log: 85.0% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T02:12:41.437 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mon.b.log: 77.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T02:12:41.437 INFO:teuthology.orchestra.run.smithi191.stderr: 90.3% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-26T02:12:41.439 INFO:teuthology.orchestra.run.smithi165.stderr: 90.2% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T02:12:41.445 INFO:teuthology.orchestra.run.smithi052.stderr: 90.2% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T02:12:41.450 INFO:teuthology.orchestra.run.smithi191.stderr: 90.2% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T02:12:41.451 INFO:teuthology.orchestra.run.smithi165.stderr: 90.1% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T02:12:41.499 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T02:12:41.501 INFO:teuthology.orchestra.run.smithi165.stderr: 92.4% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T02:12:41.532 INFO:teuthology.orchestra.run.smithi052.stderr: 88.1% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T02:12:41.654 INFO:teuthology.orchestra.run.smithi052.stderr: 91.6% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T02:12:41.738 INFO:teuthology.orchestra.run.smithi191.stderr: 94.2% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T02:12:41.739 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-26T02:12:41.739 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m0.323s 2024-03-26T02:12:41.739 INFO:teuthology.orchestra.run.smithi191.stderr:user 0m0.397s 2024-03-26T02:12:41.740 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m0.038s 2024-03-26T02:12:41.756 INFO:teuthology.orchestra.run.smithi052.stderr: 94.3% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T02:12:41.758 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-26T02:12:41.758 INFO:teuthology.orchestra.run.smithi052.stderr:real 0m0.346s 2024-03-26T02:12:41.758 INFO:teuthology.orchestra.run.smithi052.stderr:user 0m0.678s 2024-03-26T02:12:41.758 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m0.041s 2024-03-26T02:12:41.770 INFO:teuthology.orchestra.run.smithi165.stderr: 94.2% -- replaced with /var/log/ceph/46f2b2dc-eb15-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T02:12:41.773 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-03-26T02:12:41.773 INFO:teuthology.orchestra.run.smithi165.stderr:real 0m0.355s 2024-03-26T02:12:41.773 INFO:teuthology.orchestra.run.smithi165.stderr:user 0m0.426s 2024-03-26T02:12:41.773 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m0.045s 2024-03-26T02:12:41.773 INFO:tasks.cephadm:Archiving logs... 2024-03-26T02:12:41.773 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944/remote/smithi052/log 2024-03-26T02:12:41.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T02:12:41.946 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944/remote/smithi165/log 2024-03-26T02:12:41.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T02:12:42.044 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944/remote/smithi191/log 2024-03-26T02:12:42.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T02:12:42.139 INFO:tasks.cephadm:Removing cluster... 2024-03-26T02:12:42.139 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 --force 2024-03-26T02:12:42.321 INFO:teuthology.orchestra.run.smithi052.stdout:Deleting cluster with fsid: 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:12:42.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 --force 2024-03-26T02:12:42.615 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:12:42.728 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 46f2b2dc-eb15-11ee-95d0-87774f69a715 --force 2024-03-26T02:12:42.912 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 46f2b2dc-eb15-11ee-95d0-87774f69a715 2024-03-26T02:12:43.026 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T02:12:43.026 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T02:12:43.044 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T02:12:43.060 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T02:12:43.075 INFO:tasks.cephadm:Teardown complete 2024-03-26T02:12:43.075 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-26T02:12:43.085 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi052:/dev/vg_nvme/lv_1... 2024-03-26T02:12:43.086 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme disconnect -n lv_1 2024-03-26T02:12:43.242 INFO:teuthology.orchestra.run.smithi052.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T02:12:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:43.244 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi052:/dev/vg_nvme/lv_2... 2024-03-26T02:12:43.244 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme disconnect -n lv_2 2024-03-26T02:12:43.387 INFO:teuthology.orchestra.run.smithi052.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T02:12:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:43.388 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi052:/dev/vg_nvme/lv_3... 2024-03-26T02:12:43.389 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme disconnect -n lv_3 2024-03-26T02:12:43.582 INFO:teuthology.orchestra.run.smithi052.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T02:12:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:43.588 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi052:/dev/vg_nvme/lv_4... 2024-03-26T02:12:43.588 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme disconnect -n lv_4 2024-03-26T02:12:43.754 INFO:teuthology.orchestra.run.smithi052.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T02:12:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:43.756 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-26T02:12:43.756 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/scratch_devs 2024-03-26T02:12:43.782 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi165:/dev/vg_nvme/lv_1... 2024-03-26T02:12:43.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo nvme disconnect -n lv_1 2024-03-26T02:12:43.928 INFO:teuthology.orchestra.run.smithi165.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T02:12:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:43.930 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi165:/dev/vg_nvme/lv_2... 2024-03-26T02:12:43.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo nvme disconnect -n lv_2 2024-03-26T02:12:44.073 INFO:teuthology.orchestra.run.smithi165.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T02:12:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:44.075 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi165:/dev/vg_nvme/lv_3... 2024-03-26T02:12:44.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo nvme disconnect -n lv_3 2024-03-26T02:12:44.232 INFO:teuthology.orchestra.run.smithi165.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T02:12:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:44.234 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi165:/dev/vg_nvme/lv_4... 2024-03-26T02:12:44.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo nvme disconnect -n lv_4 2024-03-26T02:12:44.383 INFO:teuthology.orchestra.run.smithi165.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T02:12:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:44.385 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-26T02:12:44.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/scratch_devs 2024-03-26T02:12:44.416 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi191:/dev/vg_nvme/lv_1... 2024-03-26T02:12:44.417 DEBUG:teuthology.orchestra.run.smithi191:> sudo nvme disconnect -n lv_1 2024-03-26T02:12:44.568 INFO:teuthology.orchestra.run.smithi191.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-26T02:12:44.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:44.570 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi191:/dev/vg_nvme/lv_2... 2024-03-26T02:12:44.570 DEBUG:teuthology.orchestra.run.smithi191:> sudo nvme disconnect -n lv_2 2024-03-26T02:12:44.743 INFO:teuthology.orchestra.run.smithi191.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-26T02:12:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:44.744 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi191:/dev/vg_nvme/lv_3... 2024-03-26T02:12:44.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo nvme disconnect -n lv_3 2024-03-26T02:12:44.898 INFO:teuthology.orchestra.run.smithi191.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-26T02:12:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:44.901 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi191:/dev/vg_nvme/lv_4... 2024-03-26T02:12:44.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo nvme disconnect -n lv_4 2024-03-26T02:12:45.059 INFO:teuthology.orchestra.run.smithi191.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-26T02:12:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:45.061 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-26T02:12:45.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/scratch_devs 2024-03-26T02:12:45.089 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T02:12:45.099 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T02:12:45.100 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:12:45.102 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:12:45.104 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:12:45.115 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:12:45.119 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:12:45.120 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:12:45.120 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-03-26T02:12:45.120 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 -58us[ -48us] +/- 77ms 2024-03-26T02:12:45.120 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -402us[ -392us] +/- 58ms 2024-03-26T02:12:45.120 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +598us[ +598us] +/- 77ms 2024-03-26T02:12:45.120 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:12:45.123 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:12:45.123 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-03-26T02:12:45.123 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 2 -57us[ -57us] +/- 77ms 2024-03-26T02:12:45.123 INFO:teuthology.orchestra.run.smithi165.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -386us[ -379us] +/- 58ms 2024-03-26T02:12:45.124 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +621us[ +621us] +/- 77ms 2024-03-26T02:12:45.124 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:12:45.143 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:12:45.147 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:12:45.147 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-26T02:12:45.147 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 2 -62us[ -62us] +/- 77ms 2024-03-26T02:12:45.147 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -390us[ -381us] +/- 58ms 2024-03-26T02:12:45.147 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +611us[ +611us] +/- 77ms 2024-03-26T02:12:45.147 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:12:45.149 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T02:12:45.158 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T02:12:45.159 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T02:12:45.176 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T02:12:45.254 DEBUG:teuthology.orchestra.run.smithi165:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T02:12:45.329 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T02:12:45.399 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:12:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:45.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:12:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:45.463 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:12:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:45.492 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T02:12:45.502 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T02:12:45.543 INFO:teuthology.task.internal:Duration was 736.789747 seconds 2024-03-26T02:12:45.543 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T02:12:45.577 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T02:12:45.577 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T02:12:45.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T02:12:45.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T02:12:45.623 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:12:45.625 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:12:45.625 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:12:46.073 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T02:12:46.073 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2024-03-26T02:12:46.073 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T02:12:46.096 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-03-26T02:12:46.096 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T02:12:46.116 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-03-26T02:12:46.116 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T02:12:46.138 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T02:12:46.138 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T02:12:46.141 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T02:12:46.159 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T02:12:46.205 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T02:12:46.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T02:12:46.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T02:12:46.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T02:12:46.409 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T02:12:46.419 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T02:12:46.419 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T02:12:46.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T02:12:46.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T02:12:46.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T02:12:46.513 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:12:46.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:12:46.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:12:46.538 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2024-03-26T02:12:46.542 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-03-26T02:12:46.569 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-03-26T02:12:46.609 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:12:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:46.624 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:12:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:46.638 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:12:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:46.667 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T02:12:46.678 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T02:12:46.678 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944/remote/smithi052 2024-03-26T02:12:46.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T02:12:46.720 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944/remote/smithi165 2024-03-26T02:12:46.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T02:12:46.759 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944/remote/smithi191 2024-03-26T02:12:46.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T02:12:46.796 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T02:12:46.797 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T02:12:46.799 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T02:12:46.801 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T02:12:46.845 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T02:12:46.858 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T02:12:46.858 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T02:12:46.877 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T02:12:46.878 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T02:12:46.880 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T02:12:46.882 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T02:12:46.892 INFO:teuthology.orchestra.run.smithi052.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 02:12 /home/ubuntu/cephtest 2024-03-26T02:12:46.894 INFO:teuthology.orchestra.run.smithi165.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 02:12 /home/ubuntu/cephtest 2024-03-26T02:12:46.899 INFO:teuthology.orchestra.run.smithi191.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 02:12 /home/ubuntu/cephtest 2024-03-26T02:12:46.900 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T02:12:46.913 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T02:12:46.977 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T02:12:47.003 DEBUG:teuthology.nuke:shortname: smithi052 2024-03-26T02:12:47.004 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:12:47.024 DEBUG:teuthology.nuke:shortname: smithi165 2024-03-26T02:12:47.024 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:12:47.029 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:51:14.254090', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGpKHj+k2YawkL2g51IJ5F6/gsdHapiJ3mfoYB4MyFTcY6/WvDvTaDXCL7p9KqWiPUGpiN0syMKDa/DvuqPdiZA='} 2024-03-26T02:12:47.048 DEBUG:teuthology.nuke:shortname: smithi191 2024-03-26T02:12:47.048 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:12:47.053 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:51:14.251213', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGhgBLD76fFQ2m3Ju/nyHCweGwAPTaXT9JJBD5DN1VDc6t+HrnUijm55twL5z9hf8lKkfbkymoYZiyr1MCQAiPM='} 2024-03-26T02:12:47.084 INFO:teuthology.orchestra.console.smithi052:Power off 2024-03-26T02:12:47.084 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T02:12:47.093 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621944', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:51:14.252740', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIvncD+o5SYzAB3IgYtAx0P0I3yCWKaEQaJrn4aRIx97SFsHDXIjhiZXWGZsRoZu8GBRl4qLZhwviRop51kA70M='} 2024-03-26T02:12:47.109 DEBUG:teuthology.orchestra.console.smithi052:power off output: Chassis Power Control: Down/Off 2024-03-26T02:12:47.109 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:12:47.128 INFO:teuthology.orchestra.console.smithi165:Power off 2024-03-26T02:12:47.128 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T02:12:47.139 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-03-26T02:12:47.152 DEBUG:teuthology.orchestra.console.smithi165:power off output: Chassis Power Control: Down/Off 2024-03-26T02:12:47.152 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:12:47.168 INFO:teuthology.orchestra.console.smithi191:Power off 2024-03-26T02:12:47.168 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T02:12:47.182 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-03-26T02:12:47.190 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2024-03-26T02:12:47.190 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:12:47.214 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-26T02:12:51.140 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:12:51.183 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:12:51.215 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:12:51.250 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-03-26T02:12:51.293 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-03-26T02:12:51.324 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-26T02:12:55.254 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:12:55.295 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:12:55.325 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:12:55.366 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is off 2024-03-26T02:12:55.405 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-03-26T02:12:55.435 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2024-03-26T02:12:55.467 INFO:teuthology.orchestra.console.smithi052:Power off completed 2024-03-26T02:12:55.536 INFO:teuthology.orchestra.console.smithi191:Power off completed 2024-03-26T02:12:59.408 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:12:59.518 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is off 2024-03-26T02:12:59.619 INFO:teuthology.orchestra.console.smithi165:Power off completed 2024-03-26T02:12:59.723 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} duration: 736.7897465229034 failure_reason: '"2024-03-26T02:08:17.465556+0000 mon.a (mon.0) 249 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-26T02:12:59.723 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T02:12:59.818 INFO:teuthology.run:FAIL