2024-02-13T14:36:53.056 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-13T14:36:53.056 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_phlogistonjohn@teuthology --name phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi --description orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} --verbose --archive /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066 -- /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066/orig.config.yaml 2024-02-13T14:36:53.084 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-13T14:36:53.165 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066 branch: wip-phlogistonjohn-testing-2024-02-12-1654 description: orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} email: jmulligan@redhat.com first_in_suite: false job_id: '7558066' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-phlogistonjohn-testing-2024-02-12-1654 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-phlogistonjohn-testing-2024-02-12-1654 sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 owner: scheduled_phlogistonjohn@teuthology priority: 99 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 1588 sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 sleep_before_teardown: 0 subset: 1/15 suite: orch:cephadm suite_branch: wip-phlogistonjohn-testing-2024-02-12-1654 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e1b79d5a4193a484176863ab96c0ca28c00e1662/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 targets: smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCgujYfr5ZZ2cZPre+/pcryH2hEyo7d4eMgdZPnP5PXcPdOYXdGbRNbxjuSb7MwfEeuxj61WopXC1Jx3WlJ4qjM= smithi158.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGotlDtCMc9d/iWbkJlYPN8whM0jPDGTzY3QgD11AtZHCDqdkECOfX0jmkf4h7ZvTAak8A25jKg83wLsX6004g= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGL5pnV+NJNKGTamqUR0b5k5TOO/txtyKaQ4/YlUqj6BBU753W/lxteVYYkUGml4vzBef1TXJpo1t0sSO2/HaKI= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-13_11:41:43 tube: smithi user: phlogistonjohn verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-13T14:36:53.166 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e1b79d5a4193a484176863ab96c0ca28c00e1662/qa; will attempt to use it 2024-02-13T14:36:53.166 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e1b79d5a4193a484176863ab96c0ca28c00e1662/qa/tasks 2024-02-13T14:36:53.166 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-13T14:36:53.168 INFO:teuthology.task.internal:Checking packages... 2024-02-13T14:36:53.190 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e1b79d5a4193a484176863ab96c0ca28c00e1662' 2024-02-13T14:36:53.190 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-13T14:36:53.190 INFO:teuthology.packaging:ref: None 2024-02-13T14:36:53.190 INFO:teuthology.packaging:tag: None 2024-02-13T14:36:53.191 INFO:teuthology.packaging:branch: wip-phlogistonjohn-testing-2024-02-12-1654 2024-02-13T14:36:53.191 INFO:teuthology.packaging:sha1: e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T14:36:53.191 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-phlogistonjohn-testing-2024-02-12-1654 2024-02-13T14:36:53.433 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1423.ge1b79d5a 2024-02-13T14:36:53.435 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-13T14:36:53.441 INFO:teuthology.task.internal:no buildpackages task found 2024-02-13T14:36:53.441 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-13T14:36:53.454 INFO:teuthology.task.internal:Saving configuration 2024-02-13T14:36:53.464 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-13T14:36:53.469 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T14:36:53.495 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 14:31:02.183947', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCgujYfr5ZZ2cZPre+/pcryH2hEyo7d4eMgdZPnP5PXcPdOYXdGbRNbxjuSb7MwfEeuxj61WopXC1Jx3WlJ4qjM='} 2024-02-13T14:36:53.515 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 14:31:02.185540', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGotlDtCMc9d/iWbkJlYPN8whM0jPDGTzY3QgD11AtZHCDqdkECOfX0jmkf4h7ZvTAak8A25jKg83wLsX6004g='} 2024-02-13T14:36:53.537 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 14:31:02.186700', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGL5pnV+NJNKGTamqUR0b5k5TOO/txtyKaQ4/YlUqj6BBU753W/lxteVYYkUGml4vzBef1TXJpo1t0sSO2/HaKI='} 2024-02-13T14:36:53.537 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-13T14:36:53.543 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-13T14:36:53.543 INFO:teuthology.task.internal:roles: ubuntu@smithi158.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-02-13T14:36:53.544 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-02-13T14:36:53.544 INFO:teuthology.run_tasks:Running task console_log... 2024-02-13T14:36:53.646 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f433a403f70>, signals=[15]) 2024-02-13T14:36:53.647 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-13T14:36:53.654 INFO:teuthology.task.internal:Opening connections... 2024-02-13T14:36:53.654 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-02-13T14:36:53.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:36:53.752 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi158.front.sepia.ceph.com 2024-02-13T14:36:53.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:36:53.839 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-02-13T14:36:53.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:36:53.927 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-13T14:36:53.934 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-02-13T14:36:53.952 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-02-13T14:36:53.953 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-02-13T14:36:54.010 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-02-13T14:36:54.010 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-02-13T14:36:54.010 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-02-13T14:36:54.010 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-02-13T14:36:54.010 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-02-13T14:36:54.010 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-02-13T14:36:54.010 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-13T14:36:54.011 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-02-13T14:36:54.011 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-02-13T14:36:54.011 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-13T14:36:54.011 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-02-13T14:36:54.011 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-13T14:36:54.011 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-13T14:36:54.011 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-13T14:36:54.012 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-02-13T14:36:54.038 DEBUG:teuthology.orchestra.run.smithi158:> uname -m 2024-02-13T14:36:54.055 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64 2024-02-13T14:36:54.056 DEBUG:teuthology.orchestra.run.smithi158:> cat /etc/os-release 2024-02-13T14:36:54.112 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="CentOS Stream" 2024-02-13T14:36:54.112 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="9" 2024-02-13T14:36:54.112 INFO:teuthology.orchestra.run.smithi158.stdout:ID="centos" 2024-02-13T14:36:54.112 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE="rhel fedora" 2024-02-13T14:36:54.113 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="9" 2024-02-13T14:36:54.113 INFO:teuthology.orchestra.run.smithi158.stdout:PLATFORM_ID="platform:el9" 2024-02-13T14:36:54.113 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-13T14:36:54.113 INFO:teuthology.orchestra.run.smithi158.stdout:ANSI_COLOR="0;31" 2024-02-13T14:36:54.113 INFO:teuthology.orchestra.run.smithi158.stdout:LOGO="fedora-logo-icon" 2024-02-13T14:36:54.113 INFO:teuthology.orchestra.run.smithi158.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-13T14:36:54.113 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://centos.org/" 2024-02-13T14:36:54.113 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-13T14:36:54.113 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-13T14:36:54.113 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-13T14:36:54.113 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2024-02-13T14:36:54.137 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-02-13T14:36:54.155 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-02-13T14:36:54.156 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-02-13T14:36:54.212 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="9" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="9" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el9" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:LOGO="fedora-logo-icon" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-13T14:36:54.213 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-13T14:36:54.214 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-13T14:36:54.214 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-02-13T14:36:54.238 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-13T14:36:54.248 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-13T14:36:54.253 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-13T14:36:54.253 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-02-13T14:36:54.256 DEBUG:teuthology.orchestra.run.smithi158:> test '!' -e /home/ubuntu/cephtest 2024-02-13T14:36:54.258 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-02-13T14:36:54.274 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-13T14:36:54.281 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-13T14:36:54.281 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-02-13T14:36:54.312 DEBUG:teuthology.orchestra.run.smithi158:> test -z $(ls -A /var/lib/ceph) 2024-02-13T14:36:54.315 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-02-13T14:36:54.438 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-13T14:36:54.515 INFO:teuthology.run_tasks:Running task kernel... 2024-02-13T14:36:54.529 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-13T14:36:54.530 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-13T14:36:54.530 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-13T14:36:54.531 DEBUG:teuthology.orchestra.run.smithi158:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-13T14:36:54.531 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-13T14:36:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:36:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:36:54.548 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2024-02-13T14:36:54.549 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-02-13T14:36:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:36:54.550 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-02-13T14:36:54.605 INFO:teuthology.orchestra.run.smithi158.stdout:5.14.0-361.el9.x86_64 2024-02-13T14:36:54.605 INFO:teuthology.task.kernel:Running kernel on smithi158: 5.14.0-361.el9.x86_64 2024-02-13T14:36:54.605 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum install -y kernel 2024-02-13T14:36:54.606 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-361.el9.x86_64 2024-02-13T14:36:54.607 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.14.0-361.el9.x86_64 2024-02-13T14:36:54.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-02-13T14:36:54.608 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-361.el9.x86_64 2024-02-13T14:36:54.608 INFO:teuthology.task.kernel:Running kernel on smithi164: 5.14.0-361.el9.x86_64 2024-02-13T14:36:54.608 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum install -y kernel 2024-02-13T14:36:56.689 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-02-13T14:36:56.847 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-13T14:36:56.963 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-02-13T14:36:58.322 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-02-13T14:36:58.364 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-02-13T14:36:58.539 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-02-13T14:36:59.247 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-02-13T14:36:59.405 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-02-13T14:37:00.155 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-02-13T14:37:01.773 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-02-13T14:37:02.731 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-02-13T14:37:03.481 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:02 2024-02-13T14:37:06.774 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.9 MB 00:00 2024-02-13T14:37:06.931 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 5.9 MB 00:01 2024-02-13T14:37:08.098 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 5.9 MB 00:01 2024-02-13T14:37:08.124 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 15 kB 00:00 2024-02-13T14:37:08.732 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-02-13T14:37:09.449 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 15 kB 00:00 2024-02-13T14:37:09.875 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 20 MB/s | 20 MB 00:01 2024-02-13T14:37:12.867 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 20 MB 00:04 2024-02-13T14:37:13.511 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 5.2 MB/s | 20 MB 00:03 2024-02-13T14:37:14.808 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-02-13T14:37:17.107 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-13T14:37:17.107 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-13T14:37:17.195 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-02-13T14:37:17.199 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout:Install 4 Packages 2024-02-13T14:37:17.200 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:37:17.201 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 99 M 2024-02-13T14:37:17.201 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 122 M 2024-02-13T14:37:17.201 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-02-13T14:37:17.551 INFO:teuthology.orchestra.run.smithi164.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-02-13T14:37:17.725 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-02-13T14:37:18.151 INFO:teuthology.orchestra.run.smithi164.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-02-13T14:37:18.358 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-02-13T14:37:19.043 INFO:teuthology.orchestra.run.smithi164.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 22 MB/s | 33 MB 00:01 2024-02-13T14:37:19.493 INFO:teuthology.orchestra.run.smithi164.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 17 MB/s | 39 MB 00:02 2024-02-13T14:37:19.497 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-02-13T14:37:19.497 INFO:teuthology.orchestra.run.smithi164.stdout:Total 43 MB/s | 99 MB 00:02 2024-02-13T14:37:19.497 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-02-13T14:37:19.870 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-02-13T14:37:19.870 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-02-13T14:37:20.040 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-13T14:37:20.040 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-13T14:37:20.137 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-13T14:37:20.146 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-13T14:37:20.146 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-13T14:37:20.146 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-13T14:37:20.146 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-13T14:37:20.146 INFO:teuthology.orchestra.run.smithi158.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-13T14:37:20.146 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-13T14:37:20.146 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-13T14:37:20.146 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-13T14:37:20.147 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-13T14:37:20.147 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-13T14:37:20.147 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-13T14:37:20.147 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-13T14:37:20.147 INFO:teuthology.orchestra.run.smithi158.stdout:Install 4 Packages 2024-02-13T14:37:20.147 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-13T14:37:20.147 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 99 M 2024-02-13T14:37:20.147 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 122 M 2024-02-13T14:37:20.147 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-13T14:37:20.623 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-13T14:37:20.623 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-13T14:37:20.712 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-13T14:37:20.718 INFO:teuthology.orchestra.run.smithi158.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-02-13T14:37:20.719 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-02-13T14:37:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:37:20.721 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 99 M 2024-02-13T14:37:20.721 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 122 M 2024-02-13T14:37:20.721 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-13T14:37:20.922 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-02-13T14:37:20.923 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-02-13T14:37:21.568 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 7.1 MB/s | 6.0 MB 00:00 2024-02-13T14:37:21.660 INFO:teuthology.orchestra.run.smithi158.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 14 MB/s | 21 MB 00:01 2024-02-13T14:37:22.302 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 13 MB/s | 21 MB 00:01 2024-02-13T14:37:23.243 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-02-13T14:37:23.314 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-02-13T14:37:23.410 INFO:teuthology.orchestra.run.smithi158.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 12 MB/s | 39 MB 00:03 2024-02-13T14:37:23.726 INFO:teuthology.orchestra.run.smithi158.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 11 MB/s | 33 MB 00:03 2024-02-13T14:37:23.730 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-13T14:37:23.730 INFO:teuthology.orchestra.run.smithi158.stdout:Total 28 MB/s | 99 MB 00:03 2024-02-13T14:37:23.731 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-13T14:37:23.785 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 15 MB/s | 33 MB 00:02 2024-02-13T14:37:23.789 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-13T14:37:23.789 INFO:teuthology.orchestra.run.smithi103.stdout:Total 32 MB/s | 99 MB 00:03 2024-02-13T14:37:23.789 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-13T14:37:24.054 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-13T14:37:24.161 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-13T14:37:24.161 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-13T14:37:24.170 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-13T14:37:24.170 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-13T14:37:24.177 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:37:25.174 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-13T14:37:25.175 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-13T14:37:25.216 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-13T14:37:25.217 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-13T14:37:26.939 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:37:27.113 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:37:27.587 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-13T14:37:27.645 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-13T14:37:28.239 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-13T14:37:28.301 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-13T14:37:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:37:28.430 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:37:31.199 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:37:31.265 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:37:31.401 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:37:31.458 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:37:31.891 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:37:31.991 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:37:36.104 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:37:36.186 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:37:36.221 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:37:36.304 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:37:36.785 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:37:40.944 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:37:41.098 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:06.188 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:06.188 INFO:teuthology.orchestra.run.smithi164.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-13T14:38:06.188 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:38:06.259 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:06.654 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:06.654 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-13T14:38:06.654 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:38:06.654 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:38:07.028 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:07.028 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:38:07.028 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-02-13T14:38:07.028 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-13T14:38:07.028 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-13T14:38:07.028 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-13T14:38:07.028 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-13T14:38:07.029 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:38:07.029 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-02-13T14:38:07.558 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T14:38:07.683 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-13T14:38:07.684 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-13T14:38:07.684 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-13T14:38:07.684 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-13T14:38:07.684 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-13T14:38:07.684 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-02-13T14:38:07.684 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=None, version=distro) 2024-02-13T14:38:07.684 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T14:38:07.793 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-13T14:38:07.793 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-13T14:38:07.794 DEBUG:teuthology.orchestra.run.smithi164:> sudo rpm -qi grub2-tools 2024-02-13T14:38:07.829 INFO:teuthology.orchestra.run.smithi164.stdout:Name : grub2-tools 2024-02-13T14:38:07.829 INFO:teuthology.orchestra.run.smithi164.stdout:Epoch : 1 2024-02-13T14:38:07.829 INFO:teuthology.orchestra.run.smithi164.stdout:Version : 2.06 2024-02-13T14:38:07.829 INFO:teuthology.orchestra.run.smithi164.stdout:Release : 68.el9 2024-02-13T14:38:07.829 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: x86_64 2024-02-13T14:38:07.829 INFO:teuthology.orchestra.run.smithi164.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-13T14:38:07.829 INFO:teuthology.orchestra.run.smithi164.stdout:Group : Unspecified 2024-02-13T14:38:07.829 INFO:teuthology.orchestra.run.smithi164.stdout:Size : 8264254 2024-02-13T14:38:07.829 INFO:teuthology.orchestra.run.smithi164.stdout:License : GPLv3+ 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:Packager : builder@centos.org 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:Vendor : CentOS 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:Summary : Support tools for GRUB. 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:Description : 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout:hardware devices. 2024-02-13T14:38:07.830 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:38:07.831 INFO:teuthology.orchestra.run.smithi164.stdout:This subpackage provides tools for support of all platforms. 2024-02-13T14:38:07.831 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-13T14:38:07.831 INFO:teuthology.task.kernel:Updating grub on smithi164 to boot 5.14.0-419.el9.x86_64 2024-02-13T14:38:07.832 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-13T14:38:08.433 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2024-02-13T14:38:09.831 INFO:teuthology.orchestra.run.smithi164.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-13T14:38:09.871 INFO:teuthology.orchestra.run.smithi164.stderr:done 2024-02-13T14:38:09.873 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-02-13T14:38:09.890 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.nrO6NCAZvk 2024-02-13T14:38:09.890 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nrO6NCAZvk 2024-02-13T14:38:09.959 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.nrO6NCAZvk 2024-02-13T14:38:10.094 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.nrO6NCAZvk is 6KB 2024-02-13T14:38:10.143 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.nrO6NCAZvk 2024-02-13T14:38:10.157 DEBUG:teuthology.orchestra.run.smithi164:> sudo /bin/ls /boot/loader/entries || true 2024-02-13T14:38:10.232 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-13T14:38:10.232 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-13T14:38:10.232 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-13T14:38:10.232 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-13T14:38:10.233 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-13T14:38:10.363 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:10.363 INFO:teuthology.orchestra.run.smithi158.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-13T14:38:10.363 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-13T14:38:10.384 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2024-02-13T14:38:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:10.387 INFO:teuthology.orchestra.run.smithi103.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-13T14:38:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:38:10.434 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:10.457 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:10.879 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:10.879 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-13T14:38:10.879 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:38:10.879 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:38:10.887 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:10.888 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-13T14:38:10.888 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-13T14:38:10.888 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-13T14:38:11.308 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:11.309 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-13T14:38:11.309 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-13T14:38:11.309 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-13T14:38:11.309 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-13T14:38:11.309 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-13T14:38:11.309 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-13T14:38:11.309 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-13T14:38:11.309 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-13T14:38:11.310 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-13T14:38:11.310 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:38:11.310 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-13T14:38:11.310 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-13T14:38:11.310 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-13T14:38:11.310 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-13T14:38:11.310 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-13T14:38:11.310 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:38:11.311 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-13T14:38:11.823 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T14:38:11.853 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T14:38:11.970 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-13T14:38:11.970 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-13T14:38:11.970 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-13T14:38:11.970 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-13T14:38:11.970 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-13T14:38:11.970 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-13T14:38:11.971 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2024-02-13T14:38:11.971 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T14:38:11.995 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-13T14:38:11.995 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-13T14:38:11.995 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-13T14:38:11.995 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-13T14:38:11.995 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-13T14:38:11.995 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-13T14:38:11.995 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi158.front.sepia.ceph.com, path=None, version=distro) 2024-02-13T14:38:11.995 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2024-02-13T14:38:12.078 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-13T14:38:12.078 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-13T14:38:12.078 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-02-13T14:38:12.103 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-13T14:38:12.104 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-13T14:38:12.104 DEBUG:teuthology.orchestra.run.smithi158:> sudo rpm -qi grub2-tools 2024-02-13T14:38:12.120 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-02-13T14:38:12.120 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-02-13T14:38:12.120 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-02-13T14:38:12.120 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 68.el9 2024-02-13T14:38:12.120 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-02-13T14:38:12.120 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-13T14:38:12.120 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-02-13T14:38:12.120 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8264254 2024-02-13T14:38:12.120 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-02-13T14:38:12.120 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-13T14:38:12.120 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:38:12.121 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-02-13T14:38:12.122 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-13T14:38:12.122 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 5.14.0-419.el9.x86_64 2024-02-13T14:38:12.122 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Name : grub2-tools 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Epoch : 1 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Version : 2.06 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Release : 68.el9 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Architecture: x86_64 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Group : Unspecified 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Size : 8264254 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:License : GPLv3+ 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-13T14:38:12.142 INFO:teuthology.orchestra.run.smithi158.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout:Packager : builder@centos.org 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout:Vendor : CentOS 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout:Summary : Support tools for GRUB. 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout:Description : 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout:hardware devices. 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-13T14:38:12.143 INFO:teuthology.orchestra.run.smithi158.stdout:This subpackage provides tools for support of all platforms. 2024-02-13T14:38:12.144 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-13T14:38:12.144 INFO:teuthology.task.kernel:Updating grub on smithi158 to boot 5.14.0-419.el9.x86_64 2024-02-13T14:38:12.144 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-13T14:38:12.715 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-02-13T14:38:12.750 INFO:teuthology.orchestra.run.smithi158.stderr:Generating grub configuration file ... 2024-02-13T14:38:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-13T14:38:14.122 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-02-13T14:38:14.123 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-02-13T14:38:14.138 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.XBvH6mrN2E 2024-02-13T14:38:14.139 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XBvH6mrN2E 2024-02-13T14:38:14.207 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.XBvH6mrN2E 2024-02-13T14:38:14.208 INFO:teuthology.orchestra.run.smithi158.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-13T14:38:14.235 INFO:teuthology.orchestra.run.smithi158.stderr:done 2024-02-13T14:38:14.241 DEBUG:teuthology.orchestra.run.smithi158:> mktemp 2024-02-13T14:38:14.256 INFO:teuthology.orchestra.run.smithi158.stdout:/tmp/tmp.Ck9LdJNbE3 2024-02-13T14:38:14.257 DEBUG:teuthology.orchestra.run.smithi158:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Ck9LdJNbE3 2024-02-13T14:38:14.326 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0666 /tmp/tmp.Ck9LdJNbE3 2024-02-13T14:38:14.371 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.XBvH6mrN2E is 6KB 2024-02-13T14:38:14.419 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.XBvH6mrN2E 2024-02-13T14:38:14.433 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-02-13T14:38:14.463 DEBUG:teuthology.orchestra.remote:smithi158:/tmp/tmp.Ck9LdJNbE3 is 6KB 2024-02-13T14:38:14.497 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-13T14:38:14.497 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-13T14:38:14.497 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-13T14:38:14.497 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-13T14:38:14.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-13T14:38:14.511 DEBUG:teuthology.orchestra.run.smithi158:> rm -fr /tmp/tmp.Ck9LdJNbE3 2024-02-13T14:38:14.525 DEBUG:teuthology.orchestra.run.smithi158:> sudo /bin/ls /boot/loader/entries || true 2024-02-13T14:38:14.590 INFO:teuthology.orchestra.run.smithi158.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-13T14:38:14.590 INFO:teuthology.orchestra.run.smithi158.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-13T14:38:14.590 INFO:teuthology.orchestra.run.smithi158.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-13T14:38:14.590 INFO:teuthology.orchestra.run.smithi158.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-13T14:38:14.591 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-13T14:38:14.651 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-02-13T14:38:14.741 DEBUG:teuthology.orchestra.run.smithi158:> sudo shutdown -r now 2024-02-13T14:38:40.403 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-02-13T14:38:40.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-02-13T14:38:40.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:38:44.658 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-13T14:38:44.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-13T14:38:44.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:38:44.743 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-13T14:38:44.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-13T14:38:44.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:38:58.876 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-02-13T14:39:03.235 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-13T14:39:03.236 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-02-13T14:39:07.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-02-13T14:39:07.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:39:10.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-02-13T14:39:12.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-13T14:39:12.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:39:12.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-13T14:39:12.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:39:15.292 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-13T14:39:15.292 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-02-13T14:39:22.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-02-13T14:39:22.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:39:27.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-13T14:39:27.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:39:27.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-13T14:39:27.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:39:30.364 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-13T14:39:30.364 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-02-13T14:39:45.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-13T14:39:45.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:39:45.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-13T14:39:45.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:40:03.900 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-02-13T14:40:17.326 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-02-13T14:40:17.711 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-13T14:40:17.711 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-13T14:40:17.712 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-02-13T14:40:17.725 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-419.el9.x86_64 2024-02-13T14:40:17.725 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-13T14:40:17.725 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-13T14:40:17.726 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-13T14:40:18.726 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-13T14:40:18.726 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-13T14:40:18.780 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-02-13T14:40:18.818 DEBUG:teuthology.parallel:result is None 2024-02-13T14:40:21.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-13T14:40:21.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:40:22.239 DEBUG:teuthology.orchestra.run.smithi158:> true 2024-02-13T14:40:22.645 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-13T14:40:22.645 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-419.el9.x86_64"... 2024-02-13T14:40:22.645 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2024-02-13T14:40:22.659 INFO:teuthology.orchestra.run.smithi158.stdout:5.14.0-419.el9.x86_64 2024-02-13T14:40:22.659 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-13T14:40:22.659 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-13T14:40:22.659 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-13T14:40:22.948 DEBUG:teuthology.orchestra.remote:timed out 2024-02-13T14:40:23.661 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-13T14:40:23.661 DEBUG:teuthology.orchestra.run.smithi158:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-13T14:40:23.751 INFO:teuthology.orchestra.run.smithi158.stdout:ttyS1 2024-02-13T14:40:23.791 DEBUG:teuthology.parallel:result is None 2024-02-13T14:40:37.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-02-13T14:40:37.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:40:38.465 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-02-13T14:40:38.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-02-13T14:40:38.885 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-419.el9.x86_64"... 2024-02-13T14:40:38.885 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-02-13T14:40:38.900 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-419.el9.x86_64 2024-02-13T14:40:38.901 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-13T14:40:38.901 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-13T14:40:38.901 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-13T14:40:39.902 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-02-13T14:40:39.902 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-13T14:40:39.966 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2024-02-13T14:40:40.001 DEBUG:teuthology.parallel:result is None 2024-02-13T14:40:40.001 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-13T14:40:40.008 INFO:teuthology.task.internal:Creating test directory... 2024-02-13T14:40:40.009 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-13T14:40:40.011 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-13T14:40:40.014 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-13T14:40:40.031 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-13T14:40:40.039 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-13T14:40:40.071 INFO:teuthology.task.internal:Creating archive directory... 2024-02-13T14:40:40.071 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-13T14:40:40.073 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-13T14:40:40.076 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-13T14:40:40.111 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-13T14:40:40.117 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-13T14:40:40.118 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-13T14:40:40.137 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-13T14:40:40.148 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-13T14:40:40.177 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T14:40:40.181 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T14:40:40.184 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T14:40:40.191 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T14:40:40.194 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T14:40:40.198 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-13T14:40:40.199 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-13T14:40:40.207 INFO:teuthology.task.internal:Configuring sudo... 2024-02-13T14:40:40.207 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-13T14:40:40.209 DEBUG:teuthology.orchestra.run.smithi158:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-13T14:40:40.235 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-13T14:40:40.265 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-13T14:40:40.274 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-13T14:40:40.274 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-13T14:40:40.278 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-13T14:40:40.304 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-13T14:40:40.321 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T14:40:40.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T14:40:40.427 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T14:40:40.510 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T14:40:40.551 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:40:40.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-13T14:40:40.618 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T14:40:40.661 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T14:40:40.729 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T14:40:40.770 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T14:40:40.814 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-13T14:40:40.814 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-13T14:40:40.882 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T14:40:40.927 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-13T14:40:40.986 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T14:40:41.031 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-13T14:40:41.077 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-02-13T14:40:41.077 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-13T14:40:41.146 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-02-13T14:40:41.149 DEBUG:teuthology.orchestra.run.smithi158:> sudo service rsyslog restart 2024-02-13T14:40:41.151 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-02-13T14:40:41.184 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T14:40:41.198 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T14:40:41.231 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T14:40:41.699 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-13T14:40:41.707 INFO:teuthology.task.internal:Starting timer... 2024-02-13T14:40:41.707 INFO:teuthology.run_tasks:Running task pcp... 2024-02-13T14:40:41.716 INFO:teuthology.run_tasks:Running task selinux... 2024-02-13T14:40:41.725 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-13T14:40:41.725 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-02-13T14:40:41.811 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-02-13T14:40:41.813 DEBUG:teuthology.orchestra.run.smithi158:> sudo service auditd rotate 2024-02-13T14:40:41.872 INFO:teuthology.orchestra.run.smithi158.stdout:Rotating logs: 2024-02-13T14:40:41.874 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2024-02-13T14:40:41.928 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2024-02-13T14:40:41.929 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-13T14:40:41.930 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-02-13T14:40:41.950 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-02-13T14:40:41.950 DEBUG:teuthology.orchestra.run.smithi158:> /usr/sbin/getenforce 2024-02-13T14:40:41.979 INFO:teuthology.orchestra.run.smithi158.stdout:Permissive 2024-02-13T14:40:41.979 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2024-02-13T14:40:42.004 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2024-02-13T14:40:42.004 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi103.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi158.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive'} 2024-02-13T14:40:42.004 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T14:40:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:40:42.032 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T14:40:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:40:42.060 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T14:40:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:40:42.088 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-13T14:40:42.089 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-02-13T14:40:42.126 DEBUG:teuthology.orchestra.run.smithi158:> sudo /usr/sbin/setenforce permissive 2024-02-13T14:40:42.164 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2024-02-13T14:40:42.198 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-13T14:40:42.211 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-13T14:40:42.229 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-13T14:40:42.230 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi103.front.sepia.ceph.com,smithi158.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-13T14:44:58.424 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi103.front.sepia.ceph.com'), Remote(name='ubuntu@smithi158.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2024-02-13T14:44:58.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-13T14:44:58.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:44:58.520 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-02-13T14:44:58.604 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-13T14:44:58.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-13T14:44:58.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:44:58.690 DEBUG:teuthology.orchestra.run.smithi158:> true 2024-02-13T14:44:58.767 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-13T14:44:58.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-02-13T14:44:58.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-13T14:44:58.858 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-02-13T14:44:58.936 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-02-13T14:44:58.937 INFO:teuthology.run_tasks:Running task clock... 2024-02-13T14:44:58.953 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-13T14:44:58.953 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-13T14:44:58.954 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T14:44:58.956 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-13T14:44:58.956 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T14:44:58.959 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-13T14:44:58.959 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T14:44:58.987 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-13T14:44:58.988 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-13T14:44:59.005 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-13T14:44:59.005 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-13T14:44:59.009 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-13T14:44:59.028 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-13T14:44:59.034 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-02-13T14:44:59.035 INFO:teuthology.orchestra.run.smithi158.stderr:sudo: ntpd: command not found 2024-02-13T14:44:59.048 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-02-13T14:44:59.048 INFO:teuthology.orchestra.run.smithi158.stdout:506 Cannot talk to daemon 2024-02-13T14:44:59.058 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2024-02-13T14:44:59.064 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-13T14:44:59.065 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-13T14:44:59.072 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2024-02-13T14:44:59.081 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-13T14:44:59.081 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-13T14:44:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-13T14:44:59.105 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-13T14:44:59.133 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-02-13T14:44:59.135 INFO:teuthology.orchestra.run.smithi158.stderr:bash: line 1: ntpq: command not found 2024-02-13T14:44:59.136 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T14:44:59.136 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-02-13T14:44:59.137 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.137 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.137 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.137 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.138 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T14:44:59.138 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2024-02-13T14:44:59.138 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.138 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.138 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.138 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-02-13T14:44:59.170 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T14:44:59.170 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-02-13T14:44:59.170 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.170 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.170 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.170 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:44:59.170 INFO:teuthology.run_tasks:Running task pexec... 2024-02-13T14:44:59.181 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-13T14:44:59.182 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi103.front.sepia.ceph.com 2024-02-13T14:44:59.182 DEBUG:teuthology.orchestra.run.smithi103:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-13T14:44:59.182 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi158.front.sepia.ceph.com 2024-02-13T14:44:59.182 DEBUG:teuthology.orchestra.run.smithi158:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-13T14:44:59.183 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi164.front.sepia.ceph.com 2024-02-13T14:44:59.183 DEBUG:teuthology.orchestra.run.smithi164:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-13T14:44:59.789 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:00:50 ago on Tue 13 Feb 2024 02:44:09 PM UTC. 2024-02-13T14:44:59.791 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:50 ago on Tue 13 Feb 2024 02:44:09 PM UTC. 2024-02-13T14:44:59.819 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:51 ago on Tue 13 Feb 2024 02:44:08 PM UTC. 2024-02-13T14:44:59.942 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-13T14:44:59.947 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-13T14:44:59.948 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-13T14:44:59.948 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-02-13T14:44:59.948 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-13T14:44:59.948 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-13T14:44:59.948 INFO:teuthology.orchestra.run.smithi103.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-13T14:44:59.948 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:44:59.948 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-13T14:44:59.949 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-13T14:44:59.949 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-02-13T14:44:59.949 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:44:59.949 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 3.1 M 2024-02-13T14:44:59.949 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 9.8 M 2024-02-13T14:44:59.949 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-13T14:44:59.959 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-13T14:44:59.959 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-13T14:44:59.959 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-13T14:44:59.959 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-13T14:44:59.960 INFO:teuthology.orchestra.run.smithi158.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-13T14:44:59.960 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-13T14:44:59.960 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-13T14:44:59.960 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-13T14:44:59.960 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-13T14:44:59.960 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-13T14:44:59.960 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 3.1 M 2024-02-13T14:44:59.960 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 9.8 M 2024-02-13T14:44:59.960 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-13T14:44:59.975 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-02-13T14:44:59.984 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-02-13T14:44:59.984 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-02-13T14:44:59.984 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-02-13T14:44:59.984 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-02-13T14:44:59.984 INFO:teuthology.orchestra.run.smithi164.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-13T14:44:59.984 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:44:59.984 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-02-13T14:44:59.984 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-02-13T14:44:59.984 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-02-13T14:44:59.984 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:44:59.985 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 3.1 M 2024-02-13T14:44:59.985 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 9.8 M 2024-02-13T14:44:59.985 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-02-13T14:45:00.283 INFO:teuthology.orchestra.run.smithi103.stdout:runc-1.1.12-1.el9.x86_64.rpm 9.4 MB/s | 3.1 MB 00:00 2024-02-13T14:45:00.284 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-13T14:45:00.284 INFO:teuthology.orchestra.run.smithi103.stdout:Total 9.3 MB/s | 3.1 MB 00:00 2024-02-13T14:45:00.284 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-13T14:45:00.288 INFO:teuthology.orchestra.run.smithi158.stdout:runc-1.1.12-1.el9.x86_64.rpm 9.5 MB/s | 3.1 MB 00:00 2024-02-13T14:45:00.289 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-13T14:45:00.289 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-13T14:45:00.290 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-13T14:45:00.290 INFO:teuthology.orchestra.run.smithi158.stdout:Total 9.5 MB/s | 3.1 MB 00:00 2024-02-13T14:45:00.290 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-13T14:45:00.295 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-13T14:45:00.295 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-13T14:45:00.359 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-13T14:45:00.360 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-13T14:45:00.365 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-13T14:45:00.365 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-13T14:45:00.369 INFO:teuthology.orchestra.run.smithi164.stdout:runc-1.1.12-1.el9.x86_64.rpm 8.1 MB/s | 3.1 MB 00:00 2024-02-13T14:45:00.370 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-02-13T14:45:00.371 INFO:teuthology.orchestra.run.smithi164.stdout:Total 8.1 MB/s | 3.1 MB 00:00 2024-02-13T14:45:00.371 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-02-13T14:45:00.376 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-02-13T14:45:00.376 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-02-13T14:45:00.447 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-02-13T14:45:00.448 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-02-13T14:45:00.558 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-13T14:45:00.583 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-13T14:45:00.641 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-13T14:45:00.662 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-02-13T14:45:00.677 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-13T14:45:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-13T14:45:01.211 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-13T14:45:01.227 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-13T14:45:01.435 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-13T14:45:01.504 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-13T14:45:01.504 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-13T14:45:01.504 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-13T14:45:01.504 INFO:teuthology.orchestra.run.smithi158.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-13T14:45:01.504 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-13T14:45:01.504 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-13T14:45:01.549 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-13T14:45:01.550 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:45:01.550 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-13T14:45:01.550 INFO:teuthology.orchestra.run.smithi103.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-13T14:45:01.550 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:45:01.550 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-13T14:45:01.729 DEBUG:teuthology.parallel:result is None 2024-02-13T14:45:01.733 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-13T14:45:01.734 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:45:01.734 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-02-13T14:45:01.734 INFO:teuthology.orchestra.run.smithi164.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-13T14:45:01.734 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:45:01.734 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-02-13T14:45:01.779 DEBUG:teuthology.parallel:result is None 2024-02-13T14:45:01.960 DEBUG:teuthology.parallel:result is None 2024-02-13T14:45:01.961 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-02-13T14:45:01.971 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-02-13T14:45:01.971 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:45:01.971 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T14:45:01.986 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-13T14:45:01.987 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-02-13T14:45:02.042 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-13T14:45:02.042 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:02.042 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-13T14:45:02.042 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:02.042 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:02.042 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-13 14:44:57.351743393 +0000 2024-02-13T14:45:02.042 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-13 14:44:33.282211141 +0000 2024-02-13T14:45:02.042 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-13 14:44:33.282211141 +0000 2024-02-13T14:45:02.042 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-13 14:44:33.282211141 +0000 2024-02-13T14:45:02.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-13T14:45:02.107 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-13T14:45:02.107 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-13T14:45:02.107 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000215387 s, 2.4 MB/s 2024-02-13T14:45:02.108 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-13T14:45:02.164 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-02-13T14:45:02.220 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-13T14:45:02.220 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:02.220 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-13T14:45:02.220 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:02.220 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:02.220 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-13 14:44:57.351743393 +0000 2024-02-13T14:45:02.220 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-13 14:44:33.628204294 +0000 2024-02-13T14:45:02.220 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-13 14:44:33.628204294 +0000 2024-02-13T14:45:02.220 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-13 14:44:33.628204294 +0000 2024-02-13T14:45:02.221 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-13T14:45:02.285 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-13T14:45:02.285 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-13T14:45:02.286 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000263961 s, 1.9 MB/s 2024-02-13T14:45:02.286 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-13T14:45:02.343 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-02-13T14:45:02.398 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-13T14:45:02.399 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:02.399 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-13T14:45:02.399 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:02.399 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:02.399 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-13 14:44:57.351743393 +0000 2024-02-13T14:45:02.399 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-13 14:44:33.986197209 +0000 2024-02-13T14:45:02.399 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-13 14:44:33.986197209 +0000 2024-02-13T14:45:02.399 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-13 14:44:33.986197209 +0000 2024-02-13T14:45:02.399 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-13T14:45:02.464 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-13T14:45:02.465 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-13T14:45:02.465 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.00021287 s, 2.4 MB/s 2024-02-13T14:45:02.466 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-13T14:45:02.522 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-02-13T14:45:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-13T14:45:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:02.577 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-13T14:45:02.577 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:02.577 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:02.577 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-13 14:44:57.351743393 +0000 2024-02-13T14:45:02.578 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-13 14:44:34.336190368 +0000 2024-02-13T14:45:02.578 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-13 14:44:34.336190368 +0000 2024-02-13T14:45:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-13 14:44:34.336190368 +0000 2024-02-13T14:45:02.578 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-13T14:45:02.643 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-13T14:45:02.643 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-13T14:45:02.643 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.0002411 s, 2.1 MB/s 2024-02-13T14:45:02.644 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-13T14:45:02.700 DEBUG:teuthology.orchestra.run.smithi103:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-02-13T14:45:02.850 INFO:teuthology.orchestra.run.smithi103.stdout:loop 2024-02-13T14:45:02.851 INFO:tasks.nvme_loop:Connecting nvme_loop smithi103:/dev/vg_nvme/lv_1... 2024-02-13T14:45:02.851 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-02-13T14:45:02.890 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-02-13T14:45:02.927 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/vg_nvme/lv_11 2024-02-13T14:45:02.957 INFO:tasks.nvme_loop:Connecting nvme_loop smithi103:/dev/vg_nvme/lv_2... 2024-02-13T14:45:02.957 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-02-13T14:45:03.036 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-02-13T14:45:03.073 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/vg_nvme/lv_21 2024-02-13T14:45:03.101 INFO:tasks.nvme_loop:Connecting nvme_loop smithi103:/dev/vg_nvme/lv_3... 2024-02-13T14:45:03.101 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-02-13T14:45:03.187 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-02-13T14:45:03.226 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/vg_nvme/lv_31 2024-02-13T14:45:03.263 INFO:tasks.nvme_loop:Connecting nvme_loop smithi103:/dev/vg_nvme/lv_4... 2024-02-13T14:45:03.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-02-13T14:45:03.302 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-02-13T14:45:03.340 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/vg_nvme/lv_41 2024-02-13T14:45:03.369 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:45:03.370 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T14:45:03.425 DEBUG:teuthology.orchestra.run.smithi103:> sudo nvme list 2024-02-13T14:45:03.491 INFO:teuthology.orchestra.run.smithi103.stdout:Node SN Model Namespace Usage Format FW Rev 2024-02-13T14:45:03.492 INFO:teuthology.orchestra.run.smithi103.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-02-13T14:45:03.492 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/nvme0n1 PHFT6204012Z400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-02-13T14:45:03.492 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/nvme1n1 c4a1dd63a4557ea2e6bd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:03.492 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/nvme2n1 4908552730dcbfec68b8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:03.492 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/nvme3n1 9032664ad82a2742b35e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:03.492 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/nvme4n1 cb2988e88cbbcf4aae46 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:03.493 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-13T14:45:03.493 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:45:03.493 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/scratch_devs 2024-02-13T14:45:03.560 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-13T14:45:03.560 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T14:45:03.579 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-13T14:45:03.579 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_1 2024-02-13T14:45:03.637 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-13T14:45:03.637 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:03.638 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-13T14:45:03.638 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:03.638 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:03.638 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-02-13 14:44:57.309543551 +0000 2024-02-13T14:45:03.638 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-02-13 14:44:34.673788777 +0000 2024-02-13T14:45:03.638 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-02-13 14:44:34.673788777 +0000 2024-02-13T14:45:03.638 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-02-13 14:44:34.673788777 +0000 2024-02-13T14:45:03.638 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-02-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-02-13T14:45:03.707 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000118572 s, 4.3 MB/s 2024-02-13T14:45:03.708 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-13T14:45:03.768 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_2 2024-02-13T14:45:03.825 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-13T14:45:03.825 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:03.825 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-13T14:45:03.825 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:03.825 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:03.825 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-02-13 14:44:57.309543551 +0000 2024-02-13T14:45:03.825 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-02-13 14:44:34.999785308 +0000 2024-02-13T14:45:03.825 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-02-13 14:44:34.999785308 +0000 2024-02-13T14:45:03.825 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-02-13 14:44:34.999785308 +0000 2024-02-13T14:45:03.826 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-13T14:45:03.895 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-02-13T14:45:03.895 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-02-13T14:45:03.895 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000112586 s, 4.5 MB/s 2024-02-13T14:45:03.896 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-13T14:45:03.954 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_3 2024-02-13T14:45:04.011 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-13T14:45:04.011 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:04.011 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-13T14:45:04.011 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:04.011 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:04.011 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-02-13 14:44:57.309543551 +0000 2024-02-13T14:45:04.012 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-02-13 14:44:35.352781551 +0000 2024-02-13T14:45:04.012 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-02-13 14:44:35.352781551 +0000 2024-02-13T14:45:04.012 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-02-13 14:44:35.352781551 +0000 2024-02-13T14:45:04.012 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-13T14:45:04.078 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-02-13T14:45:04.079 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-02-13T14:45:04.079 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 8.6267e-05 s, 5.9 MB/s 2024-02-13T14:45:04.080 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-13T14:45:04.136 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_4 2024-02-13T14:45:04.192 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-13T14:45:04.192 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:04.193 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-13T14:45:04.193 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:04.193 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:04.193 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-02-13 14:44:57.309543551 +0000 2024-02-13T14:45:04.193 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-02-13 14:44:35.688777975 +0000 2024-02-13T14:45:04.193 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-02-13 14:44:35.688777975 +0000 2024-02-13T14:45:04.193 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-02-13 14:44:35.688777975 +0000 2024-02-13T14:45:04.194 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-13T14:45:04.259 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-02-13T14:45:04.260 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-02-13T14:45:04.260 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 8.404e-05 s, 6.1 MB/s 2024-02-13T14:45:04.261 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-13T14:45:04.317 DEBUG:teuthology.orchestra.run.smithi158:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-02-13T14:45:04.475 INFO:teuthology.orchestra.run.smithi158.stdout:loop 2024-02-13T14:45:04.480 INFO:tasks.nvme_loop:Connecting nvme_loop smithi158:/dev/vg_nvme/lv_1... 2024-02-13T14:45:04.480 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-02-13T14:45:04.519 INFO:teuthology.orchestra.run.smithi158.stdout:1 2024-02-13T14:45:04.556 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/vg_nvme/lv_11 2024-02-13T14:45:04.586 INFO:tasks.nvme_loop:Connecting nvme_loop smithi158:/dev/vg_nvme/lv_2... 2024-02-13T14:45:04.587 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-02-13T14:45:04.667 INFO:teuthology.orchestra.run.smithi158.stdout:1 2024-02-13T14:45:04.705 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/vg_nvme/lv_21 2024-02-13T14:45:04.733 INFO:tasks.nvme_loop:Connecting nvme_loop smithi158:/dev/vg_nvme/lv_3... 2024-02-13T14:45:04.733 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-02-13T14:45:04.813 INFO:teuthology.orchestra.run.smithi158.stdout:1 2024-02-13T14:45:04.852 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/vg_nvme/lv_31 2024-02-13T14:45:04.883 INFO:tasks.nvme_loop:Connecting nvme_loop smithi158:/dev/vg_nvme/lv_4... 2024-02-13T14:45:04.883 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-02-13T14:45:04.960 INFO:teuthology.orchestra.run.smithi158.stdout:1 2024-02-13T14:45:04.999 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/vg_nvme/lv_41 2024-02-13T14:45:05.028 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-13T14:45:05.029 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T14:45:05.084 DEBUG:teuthology.orchestra.run.smithi158:> sudo nvme list 2024-02-13T14:45:05.151 INFO:teuthology.orchestra.run.smithi158.stdout:Node SN Model Namespace Usage Format FW Rev 2024-02-13T14:45:05.151 INFO:teuthology.orchestra.run.smithi158.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-02-13T14:45:05.152 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/nvme0n1 PHMB7475005A480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-02-13T14:45:05.152 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/nvme1n1 f90377fd846a70c9952a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:05.152 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/nvme2n1 3a68e5adbaaceb9ca825 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:05.152 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/nvme3n1 4ee90acd9a7beb69896c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:05.152 INFO:teuthology.orchestra.run.smithi158.stdout:/dev/nvme4n1 8c520c98f6908f743af2 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:05.152 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-13T14:45:05.152 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-13T14:45:05.152 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/scratch_devs 2024-02-13T14:45:05.220 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-02-13T14:45:05.220 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T14:45:05.239 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-13T14:45:05.240 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-02-13T14:45:05.298 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-13T14:45:05.298 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:05.298 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-13T14:45:05.298 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:05.298 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:05.299 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-02-13 14:44:57.349721205 +0000 2024-02-13T14:45:05.299 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-02-13 14:44:34.333108948 +0000 2024-02-13T14:45:05.299 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-02-13 14:44:34.333108948 +0000 2024-02-13T14:45:05.299 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-02-13 14:44:34.333108948 +0000 2024-02-13T14:45:05.299 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-13T14:45:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-02-13T14:45:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-02-13T14:45:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000269717 s, 1.9 MB/s 2024-02-13T14:45:05.369 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-13T14:45:05.427 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-02-13T14:45:05.484 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-13T14:45:05.484 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:05.484 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-02-13T14:45:05.484 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:05.484 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:05.484 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-02-13 14:44:57.349721205 +0000 2024-02-13T14:45:05.484 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-02-13 14:44:34.671103254 +0000 2024-02-13T14:45:05.484 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-02-13 14:44:34.671103254 +0000 2024-02-13T14:45:05.484 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-02-13 14:44:34.671103254 +0000 2024-02-13T14:45:05.485 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-13T14:45:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-02-13T14:45:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-02-13T14:45:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.00021558 s, 2.4 MB/s 2024-02-13T14:45:05.553 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-13T14:45:05.611 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-02-13T14:45:05.667 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-13T14:45:05.667 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:05.667 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-13T14:45:05.667 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:05.668 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:05.668 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-02-13 14:44:57.349721205 +0000 2024-02-13T14:45:05.668 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-02-13 14:44:35.013097493 +0000 2024-02-13T14:45:05.668 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-02-13 14:44:35.013097493 +0000 2024-02-13T14:45:05.668 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-02-13 14:44:35.013097493 +0000 2024-02-13T14:45:05.668 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-13T14:45:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-02-13T14:45:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-02-13T14:45:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.00021 s, 2.4 MB/s 2024-02-13T14:45:05.736 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-13T14:45:05.793 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-02-13T14:45:05.849 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-13T14:45:05.849 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-13T14:45:05.849 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-13T14:45:05.849 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-13T14:45:05.850 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-02-13T14:45:05.850 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-02-13 14:44:57.349721205 +0000 2024-02-13T14:45:05.850 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-02-13 14:44:35.374091412 +0000 2024-02-13T14:45:05.850 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-02-13 14:44:35.374091412 +0000 2024-02-13T14:45:05.850 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-02-13 14:44:35.374091412 +0000 2024-02-13T14:45:05.850 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-13T14:45:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-02-13T14:45:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-02-13T14:45:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000198036 s, 2.6 MB/s 2024-02-13T14:45:05.920 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-13T14:45:05.978 DEBUG:teuthology.orchestra.run.smithi164:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-02-13T14:45:06.141 INFO:teuthology.orchestra.run.smithi164.stdout:loop 2024-02-13T14:45:06.142 INFO:tasks.nvme_loop:Connecting nvme_loop smithi164:/dev/vg_nvme/lv_1... 2024-02-13T14:45:06.142 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-02-13T14:45:06.180 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-02-13T14:45:06.218 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/vg_nvme/lv_11 2024-02-13T14:45:06.247 INFO:tasks.nvme_loop:Connecting nvme_loop smithi164:/dev/vg_nvme/lv_2... 2024-02-13T14:45:06.247 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-02-13T14:45:06.331 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-02-13T14:45:06.371 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/vg_nvme/lv_21 2024-02-13T14:45:06.400 INFO:tasks.nvme_loop:Connecting nvme_loop smithi164:/dev/vg_nvme/lv_3... 2024-02-13T14:45:06.400 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-02-13T14:45:06.479 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-02-13T14:45:06.518 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/vg_nvme/lv_31 2024-02-13T14:45:06.547 INFO:tasks.nvme_loop:Connecting nvme_loop smithi164:/dev/vg_nvme/lv_4... 2024-02-13T14:45:06.547 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-02-13T14:45:06.626 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-02-13T14:45:06.666 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/vg_nvme/lv_41 2024-02-13T14:45:06.707 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-02-13T14:45:06.708 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T14:45:06.764 DEBUG:teuthology.orchestra.run.smithi164:> sudo nvme list 2024-02-13T14:45:06.831 INFO:teuthology.orchestra.run.smithi164.stdout:Node SN Model Namespace Usage Format FW Rev 2024-02-13T14:45:06.831 INFO:teuthology.orchestra.run.smithi164.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-02-13T14:45:06.831 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/nvme0n1 CVFT623300H0400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-02-13T14:45:06.831 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/nvme1n1 1795a1abbc97420e2726 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:06.831 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/nvme2n1 29182662158a706d639c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:06.831 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/nvme3n1 f6fa62dbc83273c92ea1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:06.831 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/nvme4n1 abdda64967cbee926047 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-13T14:45:06.832 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-13T14:45:06.832 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-02-13T14:45:06.833 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/scratch_devs 2024-02-13T14:45:06.900 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-13T14:45:07.004 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'e1b79d5a4193a484176863ab96c0ca28c00e1662'} 2024-02-13T14:45:07.005 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T14:45:07.005 INFO:tasks.cephadm:Cluster fsid is 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:45:07.005 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-13T14:45:07.005 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.103', 'mon.b': '172.21.15.158', 'mon.c': '172.21.15.164'} 2024-02-13T14:45:07.006 INFO:tasks.cephadm:First mon is mon.a on smithi103 2024-02-13T14:45:07.006 INFO:tasks.cephadm:First mgr is y 2024-02-13T14:45:07.006 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-13T14:45:07.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo hostname $(hostname -s) 2024-02-13T14:45:07.039 DEBUG:teuthology.orchestra.run.smithi158:> sudo hostname $(hostname -s) 2024-02-13T14:45:07.073 DEBUG:teuthology.orchestra.run.smithi164:> sudo hostname $(hostname -s) 2024-02-13T14:45:07.100 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-13T14:45:07.101 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T14:45:07.241 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e1b79d5a4193a484176863ab96c0ca28c00e1662', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76678/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1423-ge1b79d5a', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1423.ge1b79d5a'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-02-12-1654/e1b79d5a4193a484176863ab96c0ca28c00e1662/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-12 23:59:26.532406', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-phlogistonjohn-testing-2024-02-12-1654', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-phlogistonjohn-testing-2024-02-12-1654/e1b79d5a4193a484176863ab96c0ca28c00e1662/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-02-13T14:45:07.512 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-phlogistonjohn-testing-2024-02-12-1654, sha1 e1b79d5a4193a484176863ab96c0ca28c00e1662 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T14:45:07.513 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-phlogistonjohn-testing-2024-02-12-1654/e1b79d5a4193a484176863ab96c0ca28c00e1662/centos/9/x86_64/flavors/default/cephadm 2024-02-13T14:45:07.514 DEBUG:teuthology.orchestra.run.smithi103:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-phlogistonjohn-testing-2024-02-12-1654/e1b79d5a4193a484176863ab96c0ca28c00e1662/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-13T14:45:08.204 INFO:teuthology.orchestra.run.smithi103.stdout:-rw-r--r--. 1 ubuntu ubuntu 779193 Feb 13 14:45 /home/ubuntu/cephtest/cephadm 2024-02-13T14:45:08.205 DEBUG:teuthology.orchestra.run.smithi158:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-phlogistonjohn-testing-2024-02-12-1654/e1b79d5a4193a484176863ab96c0ca28c00e1662/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-13T14:45:09.219 INFO:teuthology.orchestra.run.smithi158.stdout:-rw-r--r--. 1 ubuntu ubuntu 779193 Feb 13 14:45 /home/ubuntu/cephtest/cephadm 2024-02-13T14:45:09.220 DEBUG:teuthology.orchestra.run.smithi164:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-phlogistonjohn-testing-2024-02-12-1654/e1b79d5a4193a484176863ab96c0ca28c00e1662/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-13T14:45:10.553 INFO:teuthology.orchestra.run.smithi164.stdout:-rw-r--r--. 1 ubuntu ubuntu 779193 Feb 13 14:45 /home/ubuntu/cephtest/cephadm 2024-02-13T14:45:10.553 DEBUG:teuthology.orchestra.run.smithi103:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-13T14:45:10.572 DEBUG:teuthology.orchestra.run.smithi158:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-13T14:45:10.592 DEBUG:teuthology.orchestra.run.smithi164:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-13T14:45:10.624 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 on all hosts... 2024-02-13T14:45:10.627 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 pull 2024-02-13T14:45:10.630 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 pull 2024-02-13T14:45:10.634 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 pull 2024-02-13T14:45:11.414 INFO:teuthology.orchestra.run.smithi164.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662... 2024-02-13T14:45:11.442 INFO:teuthology.orchestra.run.smithi158.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662... 2024-02-13T14:45:11.503 INFO:teuthology.orchestra.run.smithi103.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662... 2024-02-13T14:45:39.129 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-02-13T14:45:39.129 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_version": "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)", 2024-02-13T14:45:39.129 INFO:teuthology.orchestra.run.smithi164.stdout: "image_id": "28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a", 2024-02-13T14:45:39.129 INFO:teuthology.orchestra.run.smithi164.stdout: "repo_digests": [ 2024-02-13T14:45:39.129 INFO:teuthology.orchestra.run.smithi164.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f" 2024-02-13T14:45:39.129 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-02-13T14:45:39.129 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-02-13T14:45:44.093 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2024-02-13T14:45:44.093 INFO:teuthology.orchestra.run.smithi158.stdout: "ceph_version": "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)", 2024-02-13T14:45:44.093 INFO:teuthology.orchestra.run.smithi158.stdout: "image_id": "28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a", 2024-02-13T14:45:44.093 INFO:teuthology.orchestra.run.smithi158.stdout: "repo_digests": [ 2024-02-13T14:45:44.093 INFO:teuthology.orchestra.run.smithi158.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f" 2024-02-13T14:45:44.094 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2024-02-13T14:45:44.094 INFO:teuthology.orchestra.run.smithi158.stdout:} 2024-02-13T14:45:44.540 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-02-13T14:45:44.540 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev)", 2024-02-13T14:45:44.540 INFO:teuthology.orchestra.run.smithi103.stdout: "image_id": "28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a", 2024-02-13T14:45:44.540 INFO:teuthology.orchestra.run.smithi103.stdout: "repo_digests": [ 2024-02-13T14:45:44.540 INFO:teuthology.orchestra.run.smithi103.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f" 2024-02-13T14:45:44.540 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-02-13T14:45:44.540 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-02-13T14:45:44.564 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph 2024-02-13T14:45:44.620 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph 2024-02-13T14:45:44.652 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph 2024-02-13T14:45:44.686 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /etc/ceph 2024-02-13T14:45:44.713 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 777 /etc/ceph 2024-02-13T14:45:44.740 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /etc/ceph 2024-02-13T14:45:44.771 INFO:tasks.cephadm:Writing seed config... 2024-02-13T14:45:44.772 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-13T14:45:44.772 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-13T14:45:44.772 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-02-13T14:45:44.772 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-02-13T14:45:44.772 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-13T14:45:44.772 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-13T14:45:44.772 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-13T14:45:44.772 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-13T14:45:44.773 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-13T14:45:44.773 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-13T14:45:44.773 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:45:44.773 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-13T14:45:44.788 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7acdb230-ca7e-11ee-95b9-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-13T14:45:44.788 DEBUG:teuthology.orchestra.run.smithi103:mon.a> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.a.service 2024-02-13T14:45:44.831 DEBUG:teuthology.orchestra.run.smithi103:mgr.y> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.y.service 2024-02-13T14:45:44.873 INFO:tasks.cephadm:Bootstrapping... 2024-02-13T14:45:44.873 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 -v bootstrap --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.103 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-13T14:45:45.067 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-13T14:45:45.067 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662', '-v', 'bootstrap', '--fsid', '7acdb230-ca7e-11ee-95b9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.103', '--skip-admin-label'] 2024-02-13T14:45:45.092 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-13T14:45:45.092 INFO:teuthology.orchestra.run.smithi103.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-13T14:45:45.093 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying podman|docker is present... 2024-02-13T14:45:45.116 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-13T14:45:45.116 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying lvm2 is present... 2024-02-13T14:45:45.116 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying time synchronization is in place... 2024-02-13T14:45:45.123 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-13T14:45:45.123 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-13T14:45:45.129 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-13T14:45:45.129 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-02-13T14:45:45.135 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout enabled 2024-02-13T14:45:45.140 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout active 2024-02-13T14:45:45.141 INFO:teuthology.orchestra.run.smithi103.stdout:Unit chronyd.service is enabled and running 2024-02-13T14:45:45.141 INFO:teuthology.orchestra.run.smithi103.stdout:Repeating the final host check... 2024-02-13T14:45:45.163 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-13T14:45:45.163 INFO:teuthology.orchestra.run.smithi103.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-13T14:45:45.163 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl is present 2024-02-13T14:45:45.163 INFO:teuthology.orchestra.run.smithi103.stdout:lvcreate is present 2024-02-13T14:45:45.170 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-13T14:45:45.170 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-13T14:45:45.175 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-13T14:45:45.175 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-02-13T14:45:45.181 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout enabled 2024-02-13T14:45:45.187 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout active 2024-02-13T14:45:45.188 INFO:teuthology.orchestra.run.smithi103.stdout:Unit chronyd.service is enabled and running 2024-02-13T14:45:45.188 INFO:teuthology.orchestra.run.smithi103.stdout:Host looks OK 2024-02-13T14:45:45.188 INFO:teuthology.orchestra.run.smithi103.stdout:Cluster fsid: 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:45:45.188 INFO:teuthology.orchestra.run.smithi103.stdout:Acquiring lock 139759509111760 on /run/cephadm/7acdb230-ca7e-11ee-95b9-87774f69a715.lock 2024-02-13T14:45:45.188 INFO:teuthology.orchestra.run.smithi103.stdout:Lock 139759509111760 acquired on /run/cephadm/7acdb230-ca7e-11ee-95b9-87774f69a715.lock 2024-02-13T14:45:45.188 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying IP 172.21.15.103 port 3300 ... 2024-02-13T14:45:45.188 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying IP 172.21.15.103 port 6789 ... 2024-02-13T14:45:45.189 INFO:teuthology.orchestra.run.smithi103.stdout:Base mon IP(s) is [172.21.15.103:3300, 172.21.15.103:6789], mon addrv is [v2:172.21.15.103:3300,v1:172.21.15.103:6789] 2024-02-13T14:45:45.192 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.103 metric 100 2024-02-13T14:45:45.192 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.103 metric 100 2024-02-13T14:45:45.194 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-13T14:45:45.194 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-13T14:45:45.194 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1764sec hoplimit 64 pref medium 2024-02-13T14:45:45.196 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-13T14:45:45.197 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-13T14:45:45.197 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-13T14:45:45.197 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-13T14:45:45.197 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-02-13T14:45:45.197 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-13T14:45:45.198 INFO:teuthology.orchestra.run.smithi103.stdout:Mon IP `172.21.15.103` is in CIDR network `172.21.0.0/20` 2024-02-13T14:45:45.198 INFO:teuthology.orchestra.run.smithi103.stdout:Mon IP `172.21.15.103` is in CIDR network `172.21.0.0/20` 2024-02-13T14:45:45.198 INFO:teuthology.orchestra.run.smithi103.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-13T14:45:45.198 INFO:teuthology.orchestra.run.smithi103.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-13T14:45:45.198 INFO:teuthology.orchestra.run.smithi103.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662... 2024-02-13T14:45:45.574 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a 2024-02-13T14:45:45.575 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662... 2024-02-13T14:45:45.575 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Getting image source signatures 2024-02-13T14:45:45.575 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying blob sha256:e42eddc1e8b8b944cc016e5274e8a770c05341ed32a5e7a2a56b9220b0323052 2024-02-13T14:45:45.575 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-13T14:45:45.575 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying config sha256:28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a 2024-02-13T14:45:45.575 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-13T14:45:46.599 INFO:teuthology.orchestra.run.smithi103.stdout:ceph: stdout ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev) 2024-02-13T14:45:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:Ceph version: ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev) 2024-02-13T14:45:46.600 INFO:teuthology.orchestra.run.smithi103.stdout:Extracting ceph user uid/gid from container image... 2024-02-13T14:45:47.460 INFO:teuthology.orchestra.run.smithi103.stdout:stat: stdout 167 167 2024-02-13T14:45:47.460 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial keys... 2024-02-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQCbgMtliOFgMBAAOIcy8rqEkM6yGzxUFU7CwQ== 2024-02-13T14:45:49.322 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQCcgMtlZt4/JxAA4JmJtqk25Tel9i22CEx7Rw== 2024-02-13T14:45:50.310 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQCdgMtldksgKRAAP4td3tnOBZLzL0q4+aDf0g== 2024-02-13T14:45:50.311 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial monmap... 2024-02-13T14:45:51.209 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-13T14:45:51.209 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-02-13T14:45:51.209 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:45:51.209 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-13T14:45:51.209 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool for a [v2:172.21.15.103:3300,v1:172.21.15.103:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-13T14:45:51.210 INFO:teuthology.orchestra.run.smithi103.stdout:setting min_mon_release = quincy 2024-02-13T14:45:51.210 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: set fsid to 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:45:51.210 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-13T14:45:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:45:51.210 INFO:teuthology.orchestra.run.smithi103.stdout:Creating mon... 2024-02-13T14:45:52.106 INFO:teuthology.orchestra.run.smithi103.stdout:create mon.a on 2024-02-13T14:45:52.448 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-13T14:45:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7acdb230-ca7e-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-7acdb230-ca7e-11ee-95b9-87774f69a715.target. 2024-02-13T14:45:52.612 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7acdb230-ca7e-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-7acdb230-ca7e-11ee-95b9-87774f69a715.target. 2024-02-13T14:45:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.a 2024-02-13T14:45:52.857 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.a.service: Unit ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.a.service not loaded. 2024-02-13T14:45:53.022 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7acdb230-ca7e-11ee-95b9-87774f69a715.target.wants/ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@.service. 2024-02-13T14:45:53.459 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:53 smithi103 podman[25324]: 2024-02-13 14:45:53.167140063 +0000 UTC m=+0.017687513 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T14:45:53.459 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:53 smithi103 podman[25324]: 2024-02-13 14:45:53.457343801 +0000 UTC m=+0.307891247 container init 8e7f6fa63cfd9cfbc8e3d214904af61ba1f643d4f4f8ec6395570ae1a928e874 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T14:45:53.593 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-13T14:45:53.593 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-02-13T14:45:53.600 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-13T14:45:53.600 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-02-13T14:45:53.601 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-02-13T14:45:53.601 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to enable service . firewalld.service is not available 2024-02-13T14:45:53.601 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mon to start... 2024-02-13T14:45:53.601 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mon... 2024-02-13T14:45:53.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:53 smithi103 podman[25324]: 2024-02-13 14:45:53.463112468 +0000 UTC m=+0.313659915 container start 8e7f6fa63cfd9cfbc8e3d214904af61ba1f643d4f4f8ec6395570ae1a928e874 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-13T14:45:53.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:53 smithi103 ceph-mon[25362]: mkfs 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:45:53.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:53 smithi103 ceph-mon[25362]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-13T14:45:53.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:53 smithi103 bash[25324]: 8e7f6fa63cfd9cfbc8e3d214904af61ba1f643d4f4f8ec6395570ae1a928e874 2024-02-13T14:45:53.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:53 smithi103 systemd[1]: Started Ceph mon.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:45:54.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:54 smithi103 ceph-mon[25362]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-13T14:45:54.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:54 smithi103 ceph-mon[25362]: monmap e1: 1 mons at {a=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T14:45:54.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:54 smithi103 ceph-mon[25362]: fsmap 2024-02-13T14:45:54.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:54 smithi103 ceph-mon[25362]: osdmap e1: 0 total, 0 up, 0 in 2024-02-13T14:45:54.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:54 smithi103 ceph-mon[25362]: mgrmap e1: no daemons active 2024-02-13T14:45:54.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:45:54 smithi103 ceph-mon[25362]: from='client.? 172.21.15.103:0/485306139' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-13T14:45:56.624 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout cluster: 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout id: 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout services: 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.557156s) 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout data: 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-13T14:45:56.625 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout pgs: 2024-02-13T14:45:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:45:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:mon is available 2024-02-13T14:45:56.626 INFO:teuthology.orchestra.run.smithi103.stdout:Assimilating anything we can from ceph.conf... 2024-02-13T14:46:00.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:00 smithi103 ceph-mon[25362]: from='client.? 172.21.15.103:0/1444261003' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-13T14:46:00.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:00 smithi103 ceph-mon[25362]: from='client.? 172.21.15.103:0/1444261003' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-13T14:46:06.344 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:46:06.344 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [global] 2024-02-13T14:46:06.344 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout fsid = 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:46:06.344 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.103:3300,v1:172.21.15.103:6789] 2024-02-13T14:46:06.344 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-13T14:46:06.344 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-13T14:46:06.344 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-13T14:46:06.344 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-13T14:46:06.344 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:46:06.344 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-13T14:46:06.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-02-13T14:46:06.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-13T14:46:06.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:46:06.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [osd] 2024-02-13T14:46:06.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-13T14:46:06.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-13T14:46:06.345 INFO:teuthology.orchestra.run.smithi103.stdout:Generating new minimal ceph.conf... 2024-02-13T14:46:07.136 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:06 smithi103 ceph-mon[25362]: from='client.? 172.21.15.103:0/2556362592' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:46:07.398 INFO:teuthology.orchestra.run.smithi103.stdout:Restarting the monitor... 2024-02-13T14:46:07.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:07 smithi103 systemd[1]: Stopping Ceph mon.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:46:08.123 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:07 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a[25339]: 2024-02-13T14:46:07.664+0000 7f763fe9c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T14:46:08.124 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:07 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a[25339]: 2024-02-13T14:46:07.664+0000 7f763fe9c700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-13T14:46:08.124 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:07 smithi103 podman[25697]: 2024-02-13 14:46:07.849742465 +0000 UTC m=+0.292552478 container died 8e7f6fa63cfd9cfbc8e3d214904af61ba1f643d4f4f8ec6395570ae1a928e874 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2) 2024-02-13T14:46:08.386 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 podman[25697]: 2024-02-13 14:46:08.204911661 +0000 UTC m=+0.647721672 container cleanup 8e7f6fa63cfd9cfbc8e3d214904af61ba1f643d4f4f8ec6395570ae1a928e874 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-13T14:46:08.386 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 bash[25697]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a 2024-02-13T14:46:08.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 podman[25716]: 2024-02-13 14:46:08.480803882 +0000 UTC m=+0.635607725 container remove 8e7f6fa63cfd9cfbc8e3d214904af61ba1f643d4f4f8ec6395570ae1a928e874 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131) 2024-02-13T14:46:08.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-13T14:46:08.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 systemd[1]: Stopped Ceph mon.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:46:08.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 systemd[1]: Starting Ceph mon.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:46:09.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 podman[25813]: 2024-02-13 14:46:08.773417139 +0000 UTC m=+0.093099482 container create cd0bedb0a109720fb060602593c8db4ae97a2cfda756aa6e467c02e9408a3dbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-13T14:46:09.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 podman[25813]: 2024-02-13 14:46:08.697755852 +0000 UTC m=+0.017438202 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T14:46:09.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 podman[25813]: 2024-02-13 14:46:08.967359892 +0000 UTC m=+0.287042249 container init cd0bedb0a109720fb060602593c8db4ae97a2cfda756aa6e467c02e9408a3dbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-13T14:46:09.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 podman[25813]: 2024-02-13 14:46:08.972985099 +0000 UTC m=+0.292667445 container start cd0bedb0a109720fb060602593c8db4ae97a2cfda756aa6e467c02e9408a3dbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a, org.label-schema.build-date=20240131, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-13T14:46:09.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 ceph-mon[25851]: set uid:gid to 167:167 (ceph:ceph) 2024-02-13T14:46:09.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 ceph-mon[25851]: ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev), process ceph-mon, pid 7 2024-02-13T14:46:09.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:08 smithi103 ceph-mon[25851]: pidfile_write: ignore empty --pid-file 2024-02-13T14:46:09.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: load: jerasure load: lrc 2024-02-13T14:46:09.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: RocksDB version: 7.9.2 2024-02-13T14:46:09.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Git sha 0 2024-02-13T14:46:09.061 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Compile date 2024-02-12 22:40:01 2024-02-13T14:46:09.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: DB SUMMARY 2024-02-13T14:46:09.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: DB Session ID: J1MV6FG1NHXTM7EQ3V5F 2024-02-13T14:46:09.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: CURRENT file: CURRENT 2024-02-13T14:46:09.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: IDENTITY file: IDENTITY 2024-02-13T14:46:09.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-13T14:46:09.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-13T14:46:09.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84651 ; 2024-02-13T14:46:09.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.error_if_exists: 0 2024-02-13T14:46:09.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.create_if_missing: 0 2024-02-13T14:46:09.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.paranoid_checks: 1 2024-02-13T14:46:09.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-13T14:46:09.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-13T14:46:09.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-13T14:46:09.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.env: 0x556ba6ba4ae0 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.fs: PosixFileSystem 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.info_log: 0x556ba751d8c0 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_file_opening_threads: 16 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.statistics: (nil) 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.use_fsync: 0 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_log_file_size: 0 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.keep_log_file_num: 1000 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.recycle_log_file_num: 0 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.allow_fallocate: 1 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.allow_mmap_reads: 0 2024-02-13T14:46:09.064 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.allow_mmap_writes: 0 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.use_direct_reads: 0 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.create_missing_column_families: 0 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.db_log_dir: 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.wal_dir: 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.advise_random_on_open: 1 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.db_write_buffer_size: 0 2024-02-13T14:46:09.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.write_buffer_manager: 0x556ba74f1400 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.rate_limiter: (nil) 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.wal_recovery_mode: 2 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.enable_thread_tracking: 0 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.enable_pipelined_write: 0 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.unordered_write: 0 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T14:46:09.066 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.row_cache: None 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.wal_filter: None 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.allow_ingest_behind: 0 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.two_write_queues: 0 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.manual_wal_flush: 0 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.wal_compression: 0 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.atomic_flush: 0 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.log_readahead_size: 0 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.best_efforts_recovery: 0 2024-02-13T14:46:09.067 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.allow_data_in_errors: 0 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.db_host_id: __hostname__ 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_background_jobs: 2 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_background_compactions: -1 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_subcompactions: 1 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_total_wal_size: 0 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T14:46:09.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_open_files: -1 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bytes_per_sync: 0 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_readahead_size: 0 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_background_flushes: -1 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Compression algorithms supported: 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: kZSTD supported: 0 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: kXpressCompression supported: 0 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: kLZ4HCCompression supported: 1 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: kZlibCompression supported: 1 2024-02-13T14:46:09.069 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: kSnappyCompression supported: 1 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: kLZ4Compression supported: 1 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: kBZip2Compression supported: 0 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.merge_operator: 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_filter: None 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_filter_factory: None 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.sst_partitioner_factory: None 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T14:46:09.070 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556ba751d9c0) 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: cache_index_and_filter_blocks: 1 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: pin_top_level_index_and_filter: 1 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: index_type: 0 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: data_block_index_type: 0 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: index_shortening: 1 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: checksum: 4 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: no_block_cache: 0 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache: 0x556ba7521090 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache_name: BinnedLRUCache 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache_options: 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: capacity : 536870912 2024-02-13T14:46:09.071 INFO:journalctl@ceph.mon.a.smithi103.stdout: num_shard_bits : 4 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: strict_capacity_limit : 0 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: high_pri_pool_ratio: 0.000 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache_compressed: (nil) 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: persistent_cache: (nil) 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_size: 4096 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_size_deviation: 10 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_restart_interval: 16 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: index_block_restart_interval: 1 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: metadata_block_size: 4096 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: partition_filters: 0 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: use_delta_encoding: 1 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: filter_policy: bloomfilter 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: whole_key_filtering: 1 2024-02-13T14:46:09.072 INFO:journalctl@ceph.mon.a.smithi103.stdout: verify_compression: 0 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout: read_amp_bytes_per_bit: 0 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout: format_version: 5 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout: enable_index_compression: 1 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_align: 0 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout: max_auto_readahead_size: 262144 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout: prepopulate_block_cache: 0 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout: initial_auto_readahead_size: 8192 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout: num_file_reads_for_auto_readahead: 2 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.write_buffer_size: 33554432 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_write_buffer_number: 2 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compression: NoCompression 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bottommost_compression: Disabled 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.prefix_extractor: nullptr 2024-02-13T14:46:09.073 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.num_levels: 7 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T14:46:09.074 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compression_opts.level: 32767 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compression_opts.strategy: 0 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compression_opts.enabled: false 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.target_file_size_base: 67108864 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T14:46:09.075 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.arena_block_size: 1048576 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T14:46:09.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.disable_auto_compactions: 0 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.table_properties_collectors: 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.inplace_update_support: 0 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T14:46:09.077 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T14:46:09.078 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T14:46:09.078 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.bloom_locality: 0 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.max_successive_merges: 0 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.paranoid_file_checks: 0 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.force_consistency_checks: 1 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.report_bg_io_stats: 0 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.ttl: 2592000 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.enable_blob_files: false 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.min_blob_size: 0 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.blob_file_size: 268435456 2024-02-13T14:46:09.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.blob_file_starting_level: 0 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dfe9163e-bc4a-4b87-a796-409e93158f41 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707835569002385, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707835569003201, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81276, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79463, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73772, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707835569, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dfe9163e-bc4a-4b87-a796-409e93158f41", "db_session_id": "J1MV6FG1NHXTM7EQ3V5F", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707835569003258, "job": 1, "event": "recovery_finished"} 2024-02-13T14:46:09.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556ba7612000 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: DB pointer 0x556ba75fc000 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** DB Stats ** 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T14:46:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** Compaction Stats [default] ** 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: L0 2/0 81.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: Sum 2/0 81.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** Compaction Stats [default] ** 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-13T14:46:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative compaction: 0.00 GB write, 27.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval compaction: 0.00 GB write, 27.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: Block cache BinnedLRUCache@0x556ba7521090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: starting mon.a rank 0 at public addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] at bind addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:46:09.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: mon.a@-1(???) e1 preinit fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: mon.a@-1(???).mds e1 new map 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: mon.a@-1(???).mds e1 print_map 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout: e1 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout: legacy client fscid: -1 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout: No filesystems configured 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-13T14:46:09.084 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 bash[25813]: cd0bedb0a109720fb060602593c8db4ae97a2cfda756aa6e467c02e9408a3dbd 2024-02-13T14:46:09.085 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 systemd[1]: Started Ceph mon.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:46:09.085 INFO:teuthology.orchestra.run.smithi103.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-13T14:46:09.318 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-13T14:46:09.318 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: monmap e1: 1 mons at {a=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T14:46:09.318 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: fsmap 2024-02-13T14:46:09.318 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: osdmap e1: 0 total, 0 up, 0 in 2024-02-13T14:46:09.318 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:09 smithi103 ceph-mon[25851]: mgrmap e1: no daemons active 2024-02-13T14:46:10.144 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-13T14:46:10.158 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-13T14:46:10.158 INFO:teuthology.orchestra.run.smithi103.stdout:Creating mgr... 2024-02-13T14:46:10.159 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-13T14:46:10.159 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-13T14:46:10.442 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.y 2024-02-13T14:46:10.449 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.y.service: Unit ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.y.service not loaded. 2024-02-13T14:46:10.591 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7acdb230-ca7e-11ee-95b9-87774f69a715.target.wants/ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@.service. 2024-02-13T14:46:10.825 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:10 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/3470758771' entity='client.admin' 2024-02-13T14:46:10.825 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:10 smithi103 systemd[1]: Starting Ceph mgr.y for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:46:11.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:10 smithi103 podman[26091]: 2024-02-13 14:46:10.725896339 +0000 UTC m=+0.018290744 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 2024-02-13T14:46:11.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:10 smithi103 podman[26091]: 2024-02-13 14:46:10.83419529 +0000 UTC m=+0.126589686 container create 10570ae595fe0a0fc95c66cd2741187d33d8a4e69c5c71abd88f72fbf20749ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-13T14:46:11.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:11 smithi103 podman[26091]: 2024-02-13 14:46:11.061702107 +0000 UTC m=+0.354096521 container init 10570ae595fe0a0fc95c66cd2741187d33d8a4e69c5c71abd88f72fbf20749ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2) 2024-02-13T14:46:11.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:11 smithi103 podman[26091]: 2024-02-13 14:46:11.067223767 +0000 UTC m=+0.359618163 container start 10570ae595fe0a0fc95c66cd2741187d33d8a4e69c5c71abd88f72fbf20749ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T14:46:11.201 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-13T14:46:11.201 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-02-13T14:46:11.208 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-13T14:46:11.208 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-02-13T14:46:11.208 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-02-13T14:46:11.208 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to enable service . firewalld.service is not available 2024-02-13T14:46:11.216 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-13T14:46:11.216 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-02-13T14:46:11.223 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-13T14:46:11.223 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-02-13T14:46:11.224 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-02-13T14:46:11.224 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-13T14:46:11.224 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr to start... 2024-02-13T14:46:11.224 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr... 2024-02-13T14:46:11.617 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:11 smithi103 bash[26091]: 10570ae595fe0a0fc95c66cd2741187d33d8a4e69c5c71abd88f72fbf20749ed 2024-02-13T14:46:11.617 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:11 smithi103 systemd[1]: Started Ceph mgr.y for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:46:11.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:11 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:11.614+0000 7f6ce3fbc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T14:46:11.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:11 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:11.709+0000 7f6ce3fbc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T14:46:12.655 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:12.352+0000 7f6ce3fbc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T14:46:12.655 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:12.444+0000 7f6ce3fbc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T14:46:12.655 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:12.601+0000 7f6ce3fbc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T14:46:12.991 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:12.701+0000 7f6ce3fbc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T14:46:12.991 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:12.783+0000 7f6ce3fbc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T14:46:13.257 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:13 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/544434022' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-13T14:46:13.257 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:13 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:13.049+0000 7f6ce3fbc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T14:46:13.257 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:13 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:13.168+0000 7f6ce3fbc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T14:46:13.257 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:13 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:13.255+0000 7f6ce3fbc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T14:46:13.530 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:13 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:13.353+0000 7f6ce3fbc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T14:46:13.531 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:13 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:13.528+0000 7f6ce3fbc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T14:46:13.743 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:46:13.744 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-02-13T14:46:13.744 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "7acdb230-ca7e-11ee-95b9-87774f69a715", 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "a" 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-02-13T14:46:13.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-13T14:46:13.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-13T14:46:13.751 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:13.751 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-13T14:46:13.751 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-13T14:46:13.751 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-13T14:46:13.752 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-13T14:46:13.752 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-13T14:46:13.752 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-13T14:46:13.752 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-13T14:46:13.752 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-13T14:46:13.752 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-13T14:46:13.752 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-13T14:46:13.753 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:13.754 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-13T14:46:13.754 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:13.754 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-02-13T14:45:53.544459+0000", 2024-02-13T14:46:13.754 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-13T14:46:13.754 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:13.754 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-13T14:46:13.754 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-02-13T14:46:13.754 INFO:teuthology.orchestra.run.smithi103.stdout:mgr not available, waiting (1/15)... 2024-02-13T14:46:14.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:14 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:14.501+0000 7f6ce3fbc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T14:46:14.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:14 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:14.630+0000 7f6ce3fbc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T14:46:15.245 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:14 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:14.960+0000 7f6ce3fbc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T14:46:15.245 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:15.049+0000 7f6ce3fbc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T14:46:15.246 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:15.243+0000 7f6ce3fbc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:46:15.654 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:15.426+0000 7f6ce3fbc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T14:46:15.972 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:15.710+0000 7f6ce3fbc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:46:15.972 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:15.970+0000 7f6ce3fbc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T14:46:16.633 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:16 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/439742772' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-13T14:46:16.633 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:16 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:16.231+0000 7f6ce3fbc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T14:46:16.897 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:46:16.897 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-02-13T14:46:16.897 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "7acdb230-ca7e-11ee-95b9-87774f69a715", 2024-02-13T14:46:16.897 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "a" 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-02-13T14:46:16.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-13T14:46:16.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:16.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-13T14:46:16.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:16.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-13T14:46:16.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:16.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-02-13T14:45:53.544459+0000", 2024-02-13T14:46:16.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-13T14:46:16.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:16.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-13T14:46:16.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-02-13T14:46:16.905 INFO:teuthology.orchestra.run.smithi103.stdout:mgr not available, waiting (2/15)... 2024-02-13T14:46:17.154 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:17.043+0000 7f6ce3fbc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T14:46:17.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:17.607+0000 7f6ce3fbc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T14:46:17.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:17.704+0000 7f6ce3fbc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T14:46:18.404 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:17.919+0000 7f6ce3fbc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T14:46:18.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: Activating manager daemon y 2024-02-13T14:46:18.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: mgrmap e2: y(active, starting, since 0.00308211s) 2024-02-13T14:46:18.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: from='mgr.14100 172.21.15.103:0/1338827542' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T14:46:18.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: from='mgr.14100 172.21.15.103:0/1338827542' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T14:46:18.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: from='mgr.14100 172.21.15.103:0/1338827542' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T14:46:18.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: from='mgr.14100 172.21.15.103:0/1338827542' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:46:18.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: from='mgr.14100 172.21.15.103:0/1338827542' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T14:46:18.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: Manager daemon y is now available 2024-02-13T14:46:18.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: from='mgr.14100 172.21.15.103:0/1338827542' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T14:46:18.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: from='mgr.14100 172.21.15.103:0/1338827542' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T14:46:18.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: from='mgr.14100 172.21.15.103:0/1338827542' entity='mgr.y' 2024-02-13T14:46:18.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: from='mgr.14100 172.21.15.103:0/1338827542' entity='mgr.y' 2024-02-13T14:46:18.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:17 smithi103 ceph-mon[25851]: from='mgr.14100 172.21.15.103:0/1338827542' entity='mgr.y' 2024-02-13T14:46:20.182 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:46:20.182 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-02-13T14:46:20.182 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "7acdb230-ca7e-11ee-95b9-87774f69a715", 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "a" 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-02-13T14:46:20.183 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-13T14:46:20.184 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-13T14:46:20.185 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-02-13T14:45:53.544459+0000", 2024-02-13T14:46:20.186 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-13T14:46:20.187 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-02-13T14:46:20.187 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-13T14:46:20.187 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-02-13T14:46:20.187 INFO:teuthology.orchestra.run.smithi103.stdout:mgr is available 2024-02-13T14:46:20.299 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:19 smithi103 ceph-mon[25851]: mgrmap e3: y(active, since 1.00657s) 2024-02-13T14:46:20.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:19 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/1787508407' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-13T14:46:21.372 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:20 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/4138058200' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-13T14:46:21.372 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:20 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/4138058200' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-13T14:46:21.376 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:46:21.376 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [global] 2024-02-13T14:46:21.376 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout fsid = 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:46:21.376 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-13T14:46:21.376 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-13T14:46:21.376 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-13T14:46:21.376 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-13T14:46:21.376 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:46:21.376 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-13T14:46:21.376 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-13T14:46:21.376 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-02-13T14:46:21.377 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [osd] 2024-02-13T14:46:21.377 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-13T14:46:21.377 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-13T14:46:21.377 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling cephadm module... 2024-02-13T14:46:22.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:21 smithi103 ceph-mon[25851]: mgrmap e4: y(active, since 3s) 2024-02-13T14:46:23.261 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:22 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/1843957877' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-13T14:46:23.261 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:22 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ignoring --setuser ceph since I am not root 2024-02-13T14:46:23.261 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:22 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ignoring --setgroup ceph since I am not root 2024-02-13T14:46:23.511 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:23 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:23.508+0000 7f87123c5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T14:46:23.785 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:23 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:23.606+0000 7f87123c5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T14:46:24.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:23 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/1843957877' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-13T14:46:24.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:23 smithi103 ceph-mon[25851]: mgrmap e5: y(active, since 5s) 2024-02-13T14:46:24.535 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:24 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:24.265+0000 7f87123c5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T14:46:24.535 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:24 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:24.365+0000 7f87123c5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T14:46:24.535 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:24 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:24.514+0000 7f87123c5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T14:46:24.893 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:24 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:24.616+0000 7f87123c5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T14:46:24.893 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:24 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:24.704+0000 7f87123c5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T14:46:24.896 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-02-13T14:46:24.896 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-13T14:46:24.896 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-13T14:46:24.896 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-13T14:46:24.897 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-13T14:46:24.897 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-02-13T14:46:24.897 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for the mgr to restart... 2024-02-13T14:46:24.897 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr epoch 5... 2024-02-13T14:46:25.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:24 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2337201194' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-13T14:46:25.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:24 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:24.980+0000 7f87123c5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T14:46:25.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:25 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:25.093+0000 7f87123c5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T14:46:25.519 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:25 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:25.200+0000 7f87123c5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T14:46:25.519 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:25 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:25.337+0000 7f87123c5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T14:46:25.519 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:25 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:25.517+0000 7f87123c5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T14:46:26.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:26 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:26.470+0000 7f87123c5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T14:46:26.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:26 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:26.599+0000 7f87123c5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T14:46:27.213 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:26 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:26.929+0000 7f87123c5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T14:46:27.213 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:27 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:27.017+0000 7f87123c5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T14:46:27.213 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:27 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:27.211+0000 7f87123c5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:46:27.654 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:27 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:27.394+0000 7f87123c5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T14:46:27.934 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:27 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:27.678+0000 7f87123c5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:46:27.934 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:27 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:27.932+0000 7f87123c5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T14:46:28.404 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:28 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:28.162+0000 7f87123c5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T14:46:29.404 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:28 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:28.944+0000 7f87123c5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T14:46:29.814 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:29.503+0000 7f87123c5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T14:46:29.814 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:29.599+0000 7f87123c5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T14:46:29.814 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:29.812+0000 7f87123c5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T14:46:30.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: Active manager daemon y restarted 2024-02-13T14:46:30.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: Activating manager daemon y 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: osdmap e2: 0 total, 0 up, 0 in 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: mgrmap e6: y(active, starting, since 0.00382369s) 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: Manager daemon y is now available 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:30.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:29 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:46:31.150 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:30 smithi103 ceph-mon[25851]: Found migration_current of "None". Setting to last migration. 2024-02-13T14:46:31.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:30 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:46:31.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:30 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:46:31.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:30 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T14:46:31.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:30 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T14:46:31.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:30 smithi103 ceph-mon[25851]: mgrmap e7: y(active, since 1.00529s) 2024-02-13T14:46:31.369 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-02-13T14:46:31.369 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-13T14:46:31.369 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-13T14:46:31.369 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-02-13T14:46:31.369 INFO:teuthology.orchestra.run.smithi103.stdout:mgr epoch 5 is available 2024-02-13T14:46:31.370 INFO:teuthology.orchestra.run.smithi103.stdout:Setting orchestrator backend to cephadm... 2024-02-13T14:46:32.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:31 smithi103 ceph-mon[25851]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-13T14:46:32.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:31 smithi103 ceph-mon[25851]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-13T14:46:32.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:31 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:32.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:31 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:32.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:31 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:46:33.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:32 smithi103 ceph-mon[25851]: [13/Feb/2024:14:46:31] ENGINE Bus STARTING 2024-02-13T14:46:33.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:32 smithi103 ceph-mon[25851]: [13/Feb/2024:14:46:31] ENGINE Serving on http://172.21.15.103:8765 2024-02-13T14:46:33.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:32 smithi103 ceph-mon[25851]: [13/Feb/2024:14:46:31] ENGINE Serving on https://172.21.15.103:7150 2024-02-13T14:46:33.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:32 smithi103 ceph-mon[25851]: [13/Feb/2024:14:46:31] ENGINE Bus STARTED 2024-02-13T14:46:33.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:32 smithi103 ceph-mon[25851]: mgrmap e8: y(active, since 2s) 2024-02-13T14:46:33.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:32 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:33.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:32 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:46:33.731 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-13T14:46:33.731 INFO:teuthology.orchestra.run.smithi103.stdout:Generating ssh key... 2024-02-13T14:46:34.123 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:33 smithi103 ceph-mon[25851]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:46:34.947 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: Generating public/private rsa key pair. 2024-02-13T14:46:34.947 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: Your identification has been saved in /tmp/tmprpa5g4kf/key. 2024-02-13T14:46:34.947 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: Your public key has been saved in /tmp/tmprpa5g4kf/key.pub. 2024-02-13T14:46:34.947 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: The key fingerprint is: 2024-02-13T14:46:34.947 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: SHA256:RWewuvmw0HrfNURIU1RAN0SHKON81gYfgU1Nl2BPh7g ceph-7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:46:34.947 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: The key's randomart image is: 2024-02-13T14:46:34.948 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: +---[RSA 3072]----+ 2024-02-13T14:46:34.948 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: | o.=%@&O| 2024-02-13T14:46:34.948 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: | .o=*oBo*| 2024-02-13T14:46:34.948 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: | ooo.=.o | 2024-02-13T14:46:34.948 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: | oo E.+ | 2024-02-13T14:46:34.948 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: | S o .. | 2024-02-13T14:46:34.948 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: | . o . | 2024-02-13T14:46:34.948 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: | . = o | 2024-02-13T14:46:34.948 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: | o.+ . . . | 2024-02-13T14:46:34.948 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: | ....o . | 2024-02-13T14:46:34.948 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: +----[SHA256]-----+ 2024-02-13T14:46:35.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-mon[25851]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:46:35.317 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:35.317 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:34 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:36.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:35 smithi103 ceph-mon[25851]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:46:36.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:35 smithi103 ceph-mon[25851]: Generating ssh key... 2024-02-13T14:46:36.369 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfIAghYzATiPy0XU8N0Upzcbk5++DxYpUQzFGshGXGXscylX32tb6pPiXcza+hNp+vLdkxL6KOd+sqIpHgESlhZVsN1t+kd7hpU7bfXAnzU/6DRIu9scdYriKMcLOAxykvedXKGX6qivQQ30LKb7m2d1ktilvbGXd9XX5UkZGi8QU3zDii18p3CIy7wZ0WivOQ1TUbcJQDk/fgTo6enoBt1+b+eNYYietbdViVbDZImpXzjTj4olVHHEDeXDVPnVVaCMbC/iDnCZYHk1/KLunwCdNLJdQPhytqwKYvodXSBQ28+VCS4znkCCDzvE75rHtOI0aeDTdehHBDXnn+jtyTCSGyDr4pXLK18trgCERxIqjU1jUKmSzsC2ssBZW2hv8NzjYPAFlfi7mtPZo5+jSCYjsVj87t0AEv4z990SSWTZiCHGo3zI9p9zGAIT5xC3gOoSB3gbhnd+JPifhaX1VXdCpzrDpOQ/67YK5GNkPqrS9VfhnkLNiiWkUIe2PtnoU= ceph-7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:46:36.369 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-13T14:46:36.369 INFO:teuthology.orchestra.run.smithi103.stdout:Adding key to root@localhost authorized_keys... 2024-02-13T14:46:36.369 INFO:teuthology.orchestra.run.smithi103.stdout:Adding host smithi103... 2024-02-13T14:46:37.068 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:36 smithi103 ceph-mon[25851]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:46:38.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:37 smithi103 ceph-mon[25851]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "addr": "172.21.15.103", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:46:39.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:38 smithi103 ceph-mon[25851]: Deploying cephadm binary to smithi103 2024-02-13T14:46:41.868 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Added host 'smithi103' with addr '172.21.15.103' 2024-02-13T14:46:41.868 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying unmanaged mon service... 2024-02-13T14:46:42.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:42 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:42.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:42 smithi103 ceph-mon[25851]: Added host smithi103 2024-02-13T14:46:42.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:42 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:46:43.436 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-13T14:46:43.437 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying unmanaged mgr service... 2024-02-13T14:46:43.771 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:43 smithi103 ceph-mon[25851]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:46:43.771 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:43 smithi103 ceph-mon[25851]: Saving service mon spec with placement count:5 2024-02-13T14:46:43.771 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:43 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:45.428 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:45 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:45.429 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:45 smithi103 ceph-mon[25851]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:46:45.429 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:45 smithi103 ceph-mon[25851]: Saving service mgr spec with placement count:2 2024-02-13T14:46:45.429 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:45 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:45.429 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:45 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:45.431 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-13T14:46:47.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:47 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/455072438' entity='client.admin' 2024-02-13T14:46:48.130 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling the dashboard module... 2024-02-13T14:46:48.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:48 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/1767438570' entity='client.admin' 2024-02-13T14:46:48.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:48 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:48.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:48 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:48.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:48 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:46:48.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:48 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' 2024-02-13T14:46:48.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:48 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi103", "caps": []}]: dispatch 2024-02-13T14:46:48.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:48 smithi103 ceph-mon[25851]: from='mgr.14118 172.21.15.103:0/1761137777' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi103", "caps": []}]': finished 2024-02-13T14:46:49.633 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:49 smithi103 ceph-mon[25851]: Deploying daemon agent.smithi103 on smithi103 2024-02-13T14:46:49.634 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:49 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/1012440529' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-13T14:46:49.634 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:49 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ignoring --setuser ceph since I am not root 2024-02-13T14:46:49.634 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:49 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ignoring --setgroup ceph since I am not root 2024-02-13T14:46:49.901 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:49 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:49.855+0000 7ff5af992200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T14:46:50.154 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:49 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:49.953+0000 7ff5af992200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T14:46:50.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:50 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/1012440529' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-13T14:46:50.615 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:50 smithi103 ceph-mon[25851]: mgrmap e9: y(active, since 19s) 2024-02-13T14:46:50.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:50 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:50.611+0000 7ff5af992200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T14:46:50.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:50 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:50.710+0000 7ff5af992200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T14:46:50.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:50 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:50.851+0000 7ff5af992200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T14:46:51.194 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-02-13T14:46:51.195 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-13T14:46:51.195 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-13T14:46:51.195 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-13T14:46:51.195 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-13T14:46:51.195 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-02-13T14:46:51.195 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for the mgr to restart... 2024-02-13T14:46:51.195 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr epoch 9... 2024-02-13T14:46:51.286 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:50 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:50.955+0000 7ff5af992200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T14:46:51.286 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:51 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:51.040+0000 7ff5af992200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T14:46:51.560 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:51 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/334701476' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-13T14:46:51.560 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:51 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:51.317+0000 7ff5af992200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T14:46:51.561 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:51 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:51.431+0000 7ff5af992200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T14:46:51.561 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:51 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:51.522+0000 7ff5af992200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T14:46:51.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:51 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:51.620+0000 7ff5af992200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T14:46:51.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:51 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:51.799+0000 7ff5af992200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T14:46:53.154 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:52 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:52.756+0000 7ff5af992200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T14:46:53.154 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:52 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:52.886+0000 7ff5af992200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T14:46:53.408 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:53 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:53.222+0000 7ff5af992200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T14:46:53.408 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:53 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:53.315+0000 7ff5af992200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T14:46:53.715 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:53 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:53.521+0000 7ff5af992200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:46:53.716 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:53 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:53.712+0000 7ff5af992200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T14:46:54.274 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:54 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:54.006+0000 7ff5af992200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:46:54.275 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:54 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:54.271+0000 7ff5af992200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T14:46:54.654 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:54 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:54.518+0000 7ff5af992200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T14:46:55.654 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:55 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:55.314+0000 7ff5af992200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T14:46:56.154 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:55 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:55.879+0000 7ff5af992200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T14:46:56.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:55 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:55.976+0000 7ff5af992200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T14:46:56.654 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:46:56.192+0000 7ff5af992200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T14:46:56.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: Active manager daemon y restarted 2024-02-13T14:46:56.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: Activating manager daemon y 2024-02-13T14:46:56.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: osdmap e3: 0 total, 0 up, 0 in 2024-02-13T14:46:56.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: mgrmap e10: y(active, starting, since 0.00308617s) 2024-02-13T14:46:56.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:46:56.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T14:46:56.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T14:46:56.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T14:46:56.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T14:46:56.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: Manager daemon y is now available 2024-02-13T14:46:56.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:46:56.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:46:56.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:56 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:46:57.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T14:46:57.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T14:46:57.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:57 smithi103 ceph-mon[25851]: mgrmap e11: y(active, since 1.00548s) 2024-02-13T14:46:57.886 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-02-13T14:46:57.886 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-13T14:46:57.886 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-13T14:46:57.886 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-02-13T14:46:57.886 INFO:teuthology.orchestra.run.smithi103.stdout:mgr epoch 9 is available 2024-02-13T14:46:57.886 INFO:teuthology.orchestra.run.smithi103.stdout:Generating a dashboard self-signed certificate... 2024-02-13T14:46:58.638 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:58 smithi103 ceph-mon[25851]: [13/Feb/2024:14:46:56] ENGINE Bus STARTING 2024-02-13T14:46:58.638 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:58 smithi103 ceph-mon[25851]: [13/Feb/2024:14:46:56] ENGINE Serving on https://172.21.15.103:7150 2024-02-13T14:46:58.638 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:58 smithi103 ceph-mon[25851]: [13/Feb/2024:14:46:57] ENGINE Serving on http://172.21.15.103:8765 2024-02-13T14:46:58.638 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:58 smithi103 ceph-mon[25851]: [13/Feb/2024:14:46:57] ENGINE Bus STARTED 2024-02-13T14:46:58.638 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:58 smithi103 ceph-mon[25851]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-13T14:46:58.638 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:58 smithi103 ceph-mon[25851]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-13T14:46:59.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:59 smithi103 ceph-mon[25851]: mgrmap e12: y(active, since 2s) 2024-02-13T14:46:59.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:46:59.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:46:59.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:46:59.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi103", "caps": []}]: dispatch 2024-02-13T14:46:59.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:46:59.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:46:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:46:59.676 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-13T14:46:59.676 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial admin user... 2024-02-13T14:47:00.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:00 smithi103 ceph-mon[25851]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:47:01.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$5bsI9rsPTThA5cQKv.pRjO.Hz9KDFKxAh8vHGKz4gRDREqOiGW4NS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707835620, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-13T14:47:01.117 INFO:teuthology.orchestra.run.smithi103.stdout:Fetching dashboard port number... 2024-02-13T14:47:01.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:01 smithi103 ceph-mon[25851]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:47:01.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:01 smithi103 ceph-mon[25851]: Deploying daemon agent.smithi103 on smithi103 2024-02-13T14:47:01.788 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:02.356 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 8443 2024-02-13T14:47:02.365 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-13T14:47:02.365 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-02-13T14:47:02.372 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-13T14:47:02.372 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-02-13T14:47:02.372 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-02-13T14:47:02.372 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-13T14:47:02.374 INFO:teuthology.orchestra.run.smithi103.stdout:Ceph Dashboard is now available at: 2024-02-13T14:47:02.374 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:02.374 INFO:teuthology.orchestra.run.smithi103.stdout: URL: https://smithi103.front.sepia.ceph.com:8443/ 2024-02-13T14:47:02.374 INFO:teuthology.orchestra.run.smithi103.stdout: User: admin 2024-02-13T14:47:02.374 INFO:teuthology.orchestra.run.smithi103.stdout: Password: weo07omuiv 2024-02-13T14:47:02.374 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:02.375 INFO:teuthology.orchestra.run.smithi103.stdout:Saving cluster configuration to /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config directory 2024-02-13T14:47:02.375 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling autotune for osd_memory_target 2024-02-13T14:47:02.605 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:02 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/699977811' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-13T14:47:03.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:03 smithi103 ceph-mon[25851]: mgrmap e13: y(active, since 6s) 2024-02-13T14:47:04.844 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-13T14:47:04.844 INFO:teuthology.orchestra.run.smithi103.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-13T14:47:04.844 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:04.844 INFO:teuthology.orchestra.run.smithi103.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-13T14:47:04.844 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:04.844 INFO:teuthology.orchestra.run.smithi103.stdout:Or, if you are only running a single cluster on this host: 2024-02-13T14:47:04.844 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:04.844 INFO:teuthology.orchestra.run.smithi103.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-13T14:47:04.844 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:04.844 INFO:teuthology.orchestra.run.smithi103.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-13T14:47:04.844 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:04.845 INFO:teuthology.orchestra.run.smithi103.stdout: ceph telemetry on 2024-02-13T14:47:04.845 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:04.845 INFO:teuthology.orchestra.run.smithi103.stdout:For more information see: 2024-02-13T14:47:04.845 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:04.845 INFO:teuthology.orchestra.run.smithi103.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-13T14:47:04.845 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:04.845 INFO:teuthology.orchestra.run.smithi103.stdout:Bootstrap complete. 2024-02-13T14:47:04.889 INFO:tasks.cephadm:Fetching config... 2024-02-13T14:47:04.890 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:47:04.890 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-13T14:47:04.905 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-13T14:47:04.906 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:47:04.906 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-13T14:47:04.960 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-13T14:47:04.960 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:47:04.960 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-13T14:47:05.124 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-13T14:47:05.125 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:47:05.125 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-13T14:47:05.186 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-13T14:47:05.186 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfIAghYzATiPy0XU8N0Upzcbk5++DxYpUQzFGshGXGXscylX32tb6pPiXcza+hNp+vLdkxL6KOd+sqIpHgESlhZVsN1t+kd7hpU7bfXAnzU/6DRIu9scdYriKMcLOAxykvedXKGX6qivQQ30LKb7m2d1ktilvbGXd9XX5UkZGi8QU3zDii18p3CIy7wZ0WivOQ1TUbcJQDk/fgTo6enoBt1+b+eNYYietbdViVbDZImpXzjTj4olVHHEDeXDVPnVVaCMbC/iDnCZYHk1/KLunwCdNLJdQPhytqwKYvodXSBQ28+VCS4znkCCDzvE75rHtOI0aeDTdehHBDXnn+jtyTCSGyDr4pXLK18trgCERxIqjU1jUKmSzsC2ssBZW2hv8NzjYPAFlfi7mtPZo5+jSCYjsVj87t0AEv4z990SSWTZiCHGo3zI9p9zGAIT5xC3gOoSB3gbhnd+JPifhaX1VXdCpzrDpOQ/67YK5GNkPqrS9VfhnkLNiiWkUIe2PtnoU= ceph-7acdb230-ca7e-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-13T14:47:05.324 INFO:teuthology.orchestra.run.smithi103.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfIAghYzATiPy0XU8N0Upzcbk5++DxYpUQzFGshGXGXscylX32tb6pPiXcza+hNp+vLdkxL6KOd+sqIpHgESlhZVsN1t+kd7hpU7bfXAnzU/6DRIu9scdYriKMcLOAxykvedXKGX6qivQQ30LKb7m2d1ktilvbGXd9XX5UkZGi8QU3zDii18p3CIy7wZ0WivOQ1TUbcJQDk/fgTo6enoBt1+b+eNYYietbdViVbDZImpXzjTj4olVHHEDeXDVPnVVaCMbC/iDnCZYHk1/KLunwCdNLJdQPhytqwKYvodXSBQ28+VCS4znkCCDzvE75rHtOI0aeDTdehHBDXnn+jtyTCSGyDr4pXLK18trgCERxIqjU1jUKmSzsC2ssBZW2hv8NzjYPAFlfi7mtPZo5+jSCYjsVj87t0AEv4z990SSWTZiCHGo3zI9p9zGAIT5xC3gOoSB3gbhnd+JPifhaX1VXdCpzrDpOQ/67YK5GNkPqrS9VfhnkLNiiWkUIe2PtnoU= ceph-7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:47:05.339 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfIAghYzATiPy0XU8N0Upzcbk5++DxYpUQzFGshGXGXscylX32tb6pPiXcza+hNp+vLdkxL6KOd+sqIpHgESlhZVsN1t+kd7hpU7bfXAnzU/6DRIu9scdYriKMcLOAxykvedXKGX6qivQQ30LKb7m2d1ktilvbGXd9XX5UkZGi8QU3zDii18p3CIy7wZ0WivOQ1TUbcJQDk/fgTo6enoBt1+b+eNYYietbdViVbDZImpXzjTj4olVHHEDeXDVPnVVaCMbC/iDnCZYHk1/KLunwCdNLJdQPhytqwKYvodXSBQ28+VCS4znkCCDzvE75rHtOI0aeDTdehHBDXnn+jtyTCSGyDr4pXLK18trgCERxIqjU1jUKmSzsC2ssBZW2hv8NzjYPAFlfi7mtPZo5+jSCYjsVj87t0AEv4z990SSWTZiCHGo3zI9p9zGAIT5xC3gOoSB3gbhnd+JPifhaX1VXdCpzrDpOQ/67YK5GNkPqrS9VfhnkLNiiWkUIe2PtnoU= ceph-7acdb230-ca7e-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-13T14:47:05.387 INFO:teuthology.orchestra.run.smithi158.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfIAghYzATiPy0XU8N0Upzcbk5++DxYpUQzFGshGXGXscylX32tb6pPiXcza+hNp+vLdkxL6KOd+sqIpHgESlhZVsN1t+kd7hpU7bfXAnzU/6DRIu9scdYriKMcLOAxykvedXKGX6qivQQ30LKb7m2d1ktilvbGXd9XX5UkZGi8QU3zDii18p3CIy7wZ0WivOQ1TUbcJQDk/fgTo6enoBt1+b+eNYYietbdViVbDZImpXzjTj4olVHHEDeXDVPnVVaCMbC/iDnCZYHk1/KLunwCdNLJdQPhytqwKYvodXSBQ28+VCS4znkCCDzvE75rHtOI0aeDTdehHBDXnn+jtyTCSGyDr4pXLK18trgCERxIqjU1jUKmSzsC2ssBZW2hv8NzjYPAFlfi7mtPZo5+jSCYjsVj87t0AEv4z990SSWTZiCHGo3zI9p9zGAIT5xC3gOoSB3gbhnd+JPifhaX1VXdCpzrDpOQ/67YK5GNkPqrS9VfhnkLNiiWkUIe2PtnoU= ceph-7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:47:05.388 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:05 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/1999900265' entity='client.admin' 2024-02-13T14:47:05.402 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfIAghYzATiPy0XU8N0Upzcbk5++DxYpUQzFGshGXGXscylX32tb6pPiXcza+hNp+vLdkxL6KOd+sqIpHgESlhZVsN1t+kd7hpU7bfXAnzU/6DRIu9scdYriKMcLOAxykvedXKGX6qivQQ30LKb7m2d1ktilvbGXd9XX5UkZGi8QU3zDii18p3CIy7wZ0WivOQ1TUbcJQDk/fgTo6enoBt1+b+eNYYietbdViVbDZImpXzjTj4olVHHEDeXDVPnVVaCMbC/iDnCZYHk1/KLunwCdNLJdQPhytqwKYvodXSBQ28+VCS4znkCCDzvE75rHtOI0aeDTdehHBDXnn+jtyTCSGyDr4pXLK18trgCERxIqjU1jUKmSzsC2ssBZW2hv8NzjYPAFlfi7mtPZo5+jSCYjsVj87t0AEv4z990SSWTZiCHGo3zI9p9zGAIT5xC3gOoSB3gbhnd+JPifhaX1VXdCpzrDpOQ/67YK5GNkPqrS9VfhnkLNiiWkUIe2PtnoU= ceph-7acdb230-ca7e-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-13T14:47:05.450 INFO:teuthology.orchestra.run.smithi164.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfIAghYzATiPy0XU8N0Upzcbk5++DxYpUQzFGshGXGXscylX32tb6pPiXcza+hNp+vLdkxL6KOd+sqIpHgESlhZVsN1t+kd7hpU7bfXAnzU/6DRIu9scdYriKMcLOAxykvedXKGX6qivQQ30LKb7m2d1ktilvbGXd9XX5UkZGi8QU3zDii18p3CIy7wZ0WivOQ1TUbcJQDk/fgTo6enoBt1+b+eNYYietbdViVbDZImpXzjTj4olVHHEDeXDVPnVVaCMbC/iDnCZYHk1/KLunwCdNLJdQPhytqwKYvodXSBQ28+VCS4znkCCDzvE75rHtOI0aeDTdehHBDXnn+jtyTCSGyDr4pXLK18trgCERxIqjU1jUKmSzsC2ssBZW2hv8NzjYPAFlfi7mtPZo5+jSCYjsVj87t0AEv4z990SSWTZiCHGo3zI9p9zGAIT5xC3gOoSB3gbhnd+JPifhaX1VXdCpzrDpOQ/67YK5GNkPqrS9VfhnkLNiiWkUIe2PtnoU= ceph-7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:47:05.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-13T14:47:05.866 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:47:07.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:07.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:07.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:07.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:07.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:07.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:07.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:07.151 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:07.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:07.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:07.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:07.152 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:08.279 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-13T14:47:08.279 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-13T14:47:08.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:08.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:08 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2988438327' entity='client.admin' 2024-02-13T14:47:08.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:08.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:09.334 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:47:10.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:10 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:10.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:10 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:12.306 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi158 2024-02-13T14:47:12.306 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-13T14:47:12.306 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/etc/ceph/ceph.conf 2024-02-13T14:47:12.325 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-13T14:47:12.325 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:47:12.383 INFO:tasks.cephadm:Adding host smithi158 to orchestrator... 2024-02-13T14:47:12.383 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch host add smithi158 2024-02-13T14:47:12.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:12 smithi103 ceph-mon[25851]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:47:12.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:12 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:12.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:12 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:12.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:12 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:12.823 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:12 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:12.824 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:12 smithi103 ceph-mon[25851]: Updating smithi103:/etc/ceph/ceph.conf 2024-02-13T14:47:12.824 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:12 smithi103 ceph-mon[25851]: Updating smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:12.824 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:12 smithi103 ceph-mon[25851]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:47:12.824 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:12 smithi103 ceph-mon[25851]: Updating smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:47:12.824 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:12 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:12.827 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:12 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:12.897 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:47:14.265 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:13 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:14.266 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:13 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:15.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:15 smithi103 ceph-mon[25851]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi158", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:47:15.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:15 smithi103 ceph-mon[25851]: Deploying cephadm binary to smithi158 2024-02-13T14:47:17.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:17 smithi103 ceph-mon[25851]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:18.821 INFO:teuthology.orchestra.run.smithi103.stdout:Added host 'smithi158' with addr '172.21.15.158' 2024-02-13T14:47:19.402 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:19 smithi103 ceph-mon[25851]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:19.402 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:19 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:19.402 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:19 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:19.430 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-13T14:47:19.666 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:47:20.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:20 smithi103 ceph-mon[25851]: Added host smithi158 2024-02-13T14:47:20.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:20 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:20.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:20 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:20.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:20 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:20.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:20 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:47:20.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:20 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:20.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:20 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:21.473 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:21.473 INFO:teuthology.orchestra.run.smithi103.stdout:[{"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}, {"addr": "172.21.15.158", "hostname": "smithi158", "labels": [], "status": ""}] 2024-02-13T14:47:21.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:21 smithi103 ceph-mon[25851]: Updating smithi158:/etc/ceph/ceph.conf 2024-02-13T14:47:21.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:21 smithi103 ceph-mon[25851]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:21.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:21 smithi103 ceph-mon[25851]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:21.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:21 smithi103 ceph-mon[25851]: Updating smithi158:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:47:21.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:21 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:21.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:21 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:21.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:21 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi158", "caps": []}]: dispatch 2024-02-13T14:47:21.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:21 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi158", "caps": []}]': finished 2024-02-13T14:47:22.010 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi164 2024-02-13T14:47:22.011 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-02-13T14:47:22.011 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/etc/ceph/ceph.conf 2024-02-13T14:47:22.030 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-02-13T14:47:22.030 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:47:22.088 INFO:tasks.cephadm:Adding host smithi164 to orchestrator... 2024-02-13T14:47:22.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch host add smithi164 2024-02-13T14:47:22.321 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:47:22.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:22 smithi103 ceph-mon[25851]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:47:23.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:47:23.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: Deploying daemon agent.smithi158 on smithi158 2024-02-13T14:47:23.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:23.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:23.608 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:23.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:23.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:23.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:23.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:23.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:23.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:23.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:23.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:23.609 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:23 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:24.887 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:24 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:24.888 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:24 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:24.888 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:24 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:24.888 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:24 smithi103 ceph-mon[25851]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi164", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:47:24.888 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:24 smithi103 ceph-mon[25851]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:26.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:25 smithi103 ceph-mon[25851]: Deploying cephadm binary to smithi164 2024-02-13T14:47:27.618 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:27 smithi103 ceph-mon[25851]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:27.618 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:27 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:27.618 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:27 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:27.618 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:27 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:28.284 INFO:teuthology.orchestra.run.smithi103.stdout:Added host 'smithi164' with addr '172.21.15.164' 2024-02-13T14:47:29.016 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:28 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:29.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:28 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:29.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:28 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:29.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:28 smithi103 ceph-mon[25851]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:29.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:28 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:29.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:28 smithi103 ceph-mon[25851]: Added host smithi164 2024-02-13T14:47:29.017 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:28 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:29.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-13T14:47:30.030 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:47:30.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:29 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:30.153 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:29 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:30.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:29 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:30.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:29 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:47:30.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:29 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:30.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:29 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:31.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:31 smithi103 ceph-mon[25851]: Updating smithi164:/etc/ceph/ceph.conf 2024-02-13T14:47:31.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:31 smithi103 ceph-mon[25851]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:31.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:31 smithi103 ceph-mon[25851]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:47:31.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:31 smithi103 ceph-mon[25851]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:47:31.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:31.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:31.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:47:31.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]': finished 2024-02-13T14:47:31.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:31 smithi103 ceph-mon[25851]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:31.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:31 smithi103 ceph-mon[25851]: Deploying daemon agent.smithi164 on smithi164 2024-02-13T14:47:31.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:32.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:32.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:32.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:32.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:32.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:32.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:32.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:32.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:32.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:32.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:32.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:32.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:32.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:32.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:32.431 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:47:32.431 INFO:teuthology.orchestra.run.smithi103.stdout:[{"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}, {"addr": "172.21.15.158", "hostname": "smithi158", "labels": [], "status": ""}, {"addr": "172.21.15.164", "hostname": "smithi164", "labels": [], "status": ""}] 2024-02-13T14:47:32.980 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-13T14:47:32.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph osd crush tunables default 2024-02-13T14:47:33.214 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:47:33.322 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:33 smithi103 ceph-mon[25851]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:34.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:34 smithi103 ceph-mon[25851]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:47:35.638 INFO:teuthology.orchestra.run.smithi103.stderr:adjusted tunables profile to default 2024-02-13T14:47:35.638 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:35 smithi103 ceph-mon[25851]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:35.638 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:35 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:35.638 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:35 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:35.638 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:35 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/185142370' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-13T14:47:36.155 INFO:tasks.cephadm:Adding mon.a on smithi103 2024-02-13T14:47:36.156 INFO:tasks.cephadm:Adding mon.b on smithi158 2024-02-13T14:47:36.156 INFO:tasks.cephadm:Adding mon.c on smithi164 2024-02-13T14:47:36.156 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch apply mon '3;smithi103:172.21.15.103=a;smithi158:172.21.15.158=b;smithi164:172.21.15.164=c' 2024-02-13T14:47:36.400 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:36.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:36 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/185142370' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-13T14:47:36.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:36 smithi103 ceph-mon[25851]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:47:36.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:36 smithi103 ceph-mon[25851]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:38.115 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled mon update... 2024-02-13T14:47:38.694 DEBUG:teuthology.orchestra.run.smithi158:mon.b> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.b.service 2024-02-13T14:47:38.697 DEBUG:teuthology.orchestra.run.smithi164:mon.c> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.c.service 2024-02-13T14:47:38.699 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-13T14:47:38.699 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T14:47:39.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:39 smithi103 ceph-mon[25851]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi103:172.21.15.103=a;smithi158:172.21.15.158=b;smithi164:172.21.15.164=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:47:39.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:39 smithi103 ceph-mon[25851]: Saving service mon spec with placement smithi103:172.21.15.103=a;smithi158:172.21.15.158=b;smithi164:172.21.15.164=c;count:3 2024-02-13T14:47:39.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:39 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:39.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:39 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:39.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:39 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:39.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:39 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:39.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:39 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:39.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:39 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:47:39.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:39 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:39.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:39 smithi103 ceph-mon[25851]: Deploying daemon mon.c on smithi164 2024-02-13T14:47:39.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:39 smithi103 ceph-mon[25851]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:39.480 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:41.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:41 smithi103 ceph-mon[25851]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:42.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:42 smithi164 podman[27576]: 2024-02-13 14:47:42.869410188 +0000 UTC m=+0.126445904 container create d2117af84018745b8725684c15641e485c18474a616719fe65b9d75a2fc0066c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True) 2024-02-13T14:47:42.967 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:47:42.967 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":1,"fsid":"7acdb230-ca7e-11ee-95b9-87774f69a715","modified":"2024-02-13T14:45:50.651665Z","created":"2024-02-13T14:45:50.651665Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-13T14:47:42.969 INFO:teuthology.orchestra.run.smithi164.stderr:dumped monmap epoch 1 2024-02-13T14:47:43.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 podman[27576]: 2024-02-13 14:47:43.248134173 +0000 UTC m=+0.505169905 container init d2117af84018745b8725684c15641e485c18474a616719fe65b9d75a2fc0066c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2) 2024-02-13T14:47:43.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 podman[27576]: 2024-02-13 14:47:43.254290545 +0000 UTC m=+0.511326261 container start d2117af84018745b8725684c15641e485c18474a616719fe65b9d75a2fc0066c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-13T14:47:43.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: set uid:gid to 167:167 (ceph:ceph) 2024-02-13T14:47:43.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev), process ceph-mon, pid 7 2024-02-13T14:47:43.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pidfile_write: ignore empty --pid-file 2024-02-13T14:47:43.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: load: jerasure load: lrc 2024-02-13T14:47:43.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: RocksDB version: 7.9.2 2024-02-13T14:47:43.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Git sha 0 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Compile date 2024-02-12 22:40:01 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: DB SUMMARY 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: DB Session ID: 3AYOIYC8G7NP10438D70 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: CURRENT file: CURRENT 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: IDENTITY file: IDENTITY 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.error_if_exists: 0 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.create_if_missing: 0 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.paranoid_checks: 1 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-13T14:47:43.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-13T14:47:43.592 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-13T14:47:43.592 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.env: 0x55dc840c4ae0 2024-02-13T14:47:43.592 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.fs: PosixFileSystem 2024-02-13T14:47:43.592 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.info_log: 0x55dc858d98c0 2024-02-13T14:47:43.592 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_file_opening_threads: 16 2024-02-13T14:47:43.593 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.statistics: (nil) 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.use_fsync: 0 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_log_file_size: 0 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.keep_log_file_num: 1000 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.recycle_log_file_num: 0 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.allow_fallocate: 1 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.allow_mmap_reads: 0 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.allow_mmap_writes: 0 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.use_direct_reads: 0 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T14:47:43.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.create_missing_column_families: 0 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.db_log_dir: 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.wal_dir: 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.advise_random_on_open: 1 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.db_write_buffer_size: 0 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.write_buffer_manager: 0x55dc858ad400 2024-02-13T14:47:43.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.rate_limiter: (nil) 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.wal_recovery_mode: 2 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.enable_thread_tracking: 0 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.enable_pipelined_write: 0 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.unordered_write: 0 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T14:47:43.596 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.row_cache: None 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.wal_filter: None 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.allow_ingest_behind: 0 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.two_write_queues: 0 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.manual_wal_flush: 0 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.wal_compression: 0 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.atomic_flush: 0 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.log_readahead_size: 0 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.best_efforts_recovery: 0 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-13T14:47:43.597 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.allow_data_in_errors: 0 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.db_host_id: __hostname__ 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_background_jobs: 2 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_background_compactions: -1 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_subcompactions: 1 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_total_wal_size: 0 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_open_files: -1 2024-02-13T14:47:43.598 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bytes_per_sync: 0 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_readahead_size: 0 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_background_flushes: -1 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Compression algorithms supported: 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: kZSTD supported: 0 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: kXpressCompression supported: 0 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: kLZ4HCCompression supported: 1 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: kZlibCompression supported: 1 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: kSnappyCompression supported: 1 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: kLZ4Compression supported: 1 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: kBZip2Compression supported: 0 2024-02-13T14:47:43.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.merge_operator: 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_filter: None 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_filter_factory: None 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.sst_partitioner_factory: None 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dc858d99c0) 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout: cache_index_and_filter_blocks: 1 2024-02-13T14:47:43.600 INFO:journalctl@ceph.mon.c.smithi164.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: pin_top_level_index_and_filter: 1 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: index_type: 0 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: data_block_index_type: 0 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: index_shortening: 1 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: checksum: 4 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: no_block_cache: 0 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache: 0x55dc858dd090 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache_name: BinnedLRUCache 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache_options: 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: capacity : 536870912 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: num_shard_bits : 4 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: strict_capacity_limit : 0 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: high_pri_pool_ratio: 0.000 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache_compressed: (nil) 2024-02-13T14:47:43.601 INFO:journalctl@ceph.mon.c.smithi164.stdout: persistent_cache: (nil) 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_size: 4096 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_size_deviation: 10 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_restart_interval: 16 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: index_block_restart_interval: 1 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: metadata_block_size: 4096 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: partition_filters: 0 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: use_delta_encoding: 1 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: filter_policy: bloomfilter 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: whole_key_filtering: 1 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: verify_compression: 0 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: read_amp_bytes_per_bit: 0 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: format_version: 5 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: enable_index_compression: 1 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_align: 0 2024-02-13T14:47:43.602 INFO:journalctl@ceph.mon.c.smithi164.stdout: max_auto_readahead_size: 262144 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout: prepopulate_block_cache: 0 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout: initial_auto_readahead_size: 8192 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout: num_file_reads_for_auto_readahead: 2 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.write_buffer_size: 33554432 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_write_buffer_number: 2 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compression: NoCompression 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bottommost_compression: Disabled 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.prefix_extractor: nullptr 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.num_levels: 7 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T14:47:43.603 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compression_opts.level: 32767 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compression_opts.strategy: 0 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-13T14:47:43.604 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compression_opts.enabled: false 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.target_file_size_base: 67108864 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T14:47:43.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.arena_block_size: 1048576 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.disable_auto_compactions: 0 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T14:47:43.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.table_properties_collectors: 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.inplace_update_support: 0 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.bloom_locality: 0 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.max_successive_merges: 0 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.paranoid_file_checks: 0 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.force_consistency_checks: 1 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.report_bg_io_stats: 0 2024-02-13T14:47:43.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.ttl: 2592000 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.enable_blob_files: false 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.min_blob_size: 0 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.blob_file_size: 268435456 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.blob_file_starting_level: 0 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-13T14:47:43.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f0e22944-d9ae-4920-b22c-525207a92a07 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707835663284680, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707835663285255, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707835663, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f0e22944-d9ae-4920-b22c-525207a92a07", "db_session_id": "3AYOIYC8G7NP10438D70", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707835663285362, "job": 1, "event": "recovery_finished"} 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dc859ce000 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: DB pointer 0x55dc859b8000 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** DB Stats ** 2024-02-13T14:47:43.609 INFO:journalctl@ceph.mon.c.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** Compaction Stats [default] ** 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-02-13T14:47:43.610 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** Compaction Stats [default] ** 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T14:47:43.611 INFO:journalctl@ceph.mon.c.smithi164.stdout: Block cache BinnedLRUCache@0x55dc858dd090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: using public_addr v2:172.21.15.164:0/0 -> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: starting mon.c rank -1 at public addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] at bind addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(???) e0 preinit fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).mds e1 new map 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout: e1 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout: legacy client fscid: -1 2024-02-13T14:47:43.612 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout: No filesystems configured 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Added host smithi158 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.613 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Updating smithi158:/etc/ceph/ceph.conf 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Updating smithi158:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi158", "caps": []}]: dispatch 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi158", "caps": []}]': finished 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Deploying daemon agent.smithi158 on smithi158 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi164", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:47:43.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Deploying cephadm binary to smithi164 2024-02-13T14:47:43.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Added host smithi164 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Updating smithi164:/etc/ceph/ceph.conf 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]': finished 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Deploying daemon agent.smithi164 on smithi164 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.618 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/185142370' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/185142370' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi103:172.21.15.103=a;smithi158:172.21.15.158=b;smithi164:172.21.15.164=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Saving service mon spec with placement smithi103:172.21.15.103=a;smithi158:172.21.15.158=b;smithi164:172.21.15.164=c;count:3 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.619 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:43.620 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:43.620 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:43.620 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:43.620 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:47:43.620 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:43.620 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: Deploying daemon mon.c on smithi164 2024-02-13T14:47:43.620 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.620 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.620 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.620 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: from='client.? 172.21.15.164:0/944558291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T14:47:43.620 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 ceph-mon[27636]: mon.c@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-02-13T14:47:43.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:43 smithi103 ceph-mon[25851]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:43.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:43 smithi103 ceph-mon[25851]: from='client.? 172.21.15.164:0/944558291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T14:47:43.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 bash[27576]: d2117af84018745b8725684c15641e485c18474a616719fe65b9d75a2fc0066c 2024-02-13T14:47:43.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:43 smithi164 systemd[1]: Started Ceph mon.c for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:47:44.622 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-13T14:47:44.622 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T14:47:44.858 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.c/config 2024-02-13T14:47:48.612 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: mon.a calling monitor election 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: mon.c calling monitor election 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.613 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: monmap e2: 2 mons at {a=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: fsmap 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: mgrmap e13: y(active, since 52s) 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: overall HEALTH_OK 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.614 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.615 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.615 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:47:48.615 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:48.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:47:48.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: mon.a calling monitor election 2024-02-13T14:47:48.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:48.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: mon.c calling monitor election 2024-02-13T14:47:48.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:48.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: monmap e2: 2 mons at {a=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: fsmap 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: mgrmap e13: y(active, since 52s) 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: overall HEALTH_OK 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:48.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:47:48.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:49.613 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:47:49.613 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":2,"fsid":"7acdb230-ca7e-11ee-95b9-87774f69a715","modified":"2024-02-13T14:47:43.311593Z","created":"2024-02-13T14:45:50.651665Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-13T14:47:49.615 INFO:teuthology.orchestra.run.smithi164.stderr:dumped monmap epoch 2 2024-02-13T14:47:49.757 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:49 smithi103 ceph-mon[25851]: Deploying daemon mon.b on smithi158 2024-02-13T14:47:49.758 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:49 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:49.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:49 smithi164 ceph-mon[27636]: Deploying daemon mon.b on smithi158 2024-02-13T14:47:49.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:49 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:50.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:50 smithi164 ceph-mon[27636]: from='client.? 172.21.15.164:0/2046542432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T14:47:50.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:50 smithi164 ceph-mon[27636]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:50.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:50 smithi103 ceph-mon[25851]: from='client.? 172.21.15.164:0/2046542432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T14:47:50.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:50 smithi103 ceph-mon[25851]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:51.216 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-13T14:47:51.217 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-13T14:47:51.461 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.c/config 2024-02-13T14:47:52.342 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 podman[27591]: 2024-02-13 14:47:52.340577411 +0000 UTC m=+0.289196567 container init cd1f74efc812685ac576d27bf6410246a64c1421ee9aa0df6d6f26119b587f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-13T14:47:52.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 podman[27591]: 2024-02-13 14:47:52.346232675 +0000 UTC m=+0.294851829 container start cd1f74efc812685ac576d27bf6410246a64c1421ee9aa0df6d6f26119b587f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-b, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-13T14:47:52.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: set uid:gid to 167:167 (ceph:ceph) 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: ceph version 19.0.0-1423-ge1b79d5a (e1b79d5a4193a484176863ab96c0ca28c00e1662) squid (dev), process ceph-mon, pid 7 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pidfile_write: ignore empty --pid-file 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: load: jerasure load: lrc 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: RocksDB version: 7.9.2 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Git sha 0 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Compile date 2024-02-12 22:40:01 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: DB SUMMARY 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: DB Session ID: WGD14P953EZQRKKO5N87 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: CURRENT file: CURRENT 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: IDENTITY file: IDENTITY 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-02-13T14:47:52.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.error_if_exists: 0 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.create_if_missing: 0 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.paranoid_checks: 1 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.env: 0x55ecaf2c3ae0 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.fs: PosixFileSystem 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.info_log: 0x55ecb10a78c0 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_file_opening_threads: 16 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.statistics: (nil) 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.use_fsync: 0 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_log_file_size: 0 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-13T14:47:52.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.keep_log_file_num: 1000 2024-02-13T14:47:52.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.recycle_log_file_num: 0 2024-02-13T14:47:52.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.allow_fallocate: 1 2024-02-13T14:47:52.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.allow_mmap_reads: 0 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.allow_mmap_writes: 0 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.use_direct_reads: 0 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.create_missing_column_families: 0 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.db_log_dir: 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.wal_dir: 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-13T14:47:52.683 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.advise_random_on_open: 1 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.db_write_buffer_size: 0 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.write_buffer_manager: 0x55ecb107b400 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.rate_limiter: (nil) 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.wal_recovery_mode: 2 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.enable_thread_tracking: 0 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.enable_pipelined_write: 0 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.unordered_write: 0 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-13T14:47:52.684 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.row_cache: None 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.wal_filter: None 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.allow_ingest_behind: 0 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.two_write_queues: 0 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.manual_wal_flush: 0 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.wal_compression: 0 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.atomic_flush: 0 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.log_readahead_size: 0 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-13T14:47:52.685 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.best_efforts_recovery: 0 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.allow_data_in_errors: 0 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.db_host_id: __hostname__ 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_background_jobs: 2 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_background_compactions: -1 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_subcompactions: 1 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_total_wal_size: 0 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-13T14:47:52.686 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_open_files: -1 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bytes_per_sync: 0 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_readahead_size: 0 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_background_flushes: -1 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Compression algorithms supported: 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: kZSTD supported: 0 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: kXpressCompression supported: 0 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: kLZ4HCCompression supported: 1 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: kZlibCompression supported: 1 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: kSnappyCompression supported: 1 2024-02-13T14:47:52.687 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: kLZ4Compression supported: 1 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: kBZip2Compression supported: 0 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.merge_operator: 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_filter: None 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_filter_factory: None 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.sst_partitioner_factory: None 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ecb10a79c0) 2024-02-13T14:47:52.688 INFO:journalctl@ceph.mon.b.smithi158.stdout: cache_index_and_filter_blocks: 1 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: pin_top_level_index_and_filter: 1 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: index_type: 0 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: data_block_index_type: 0 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: index_shortening: 1 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: checksum: 4 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: no_block_cache: 0 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_cache: 0x55ecb10ab090 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_cache_name: BinnedLRUCache 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_cache_options: 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: capacity : 536870912 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: num_shard_bits : 4 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: strict_capacity_limit : 0 2024-02-13T14:47:52.689 INFO:journalctl@ceph.mon.b.smithi158.stdout: high_pri_pool_ratio: 0.000 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_cache_compressed: (nil) 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: persistent_cache: (nil) 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_size: 4096 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_size_deviation: 10 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_restart_interval: 16 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: index_block_restart_interval: 1 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: metadata_block_size: 4096 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: partition_filters: 0 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: use_delta_encoding: 1 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: filter_policy: bloomfilter 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: whole_key_filtering: 1 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: verify_compression: 0 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: read_amp_bytes_per_bit: 0 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: format_version: 5 2024-02-13T14:47:52.690 INFO:journalctl@ceph.mon.b.smithi158.stdout: enable_index_compression: 1 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout: block_align: 0 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout: max_auto_readahead_size: 262144 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout: prepopulate_block_cache: 0 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout: initial_auto_readahead_size: 8192 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout: num_file_reads_for_auto_readahead: 2 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.write_buffer_size: 33554432 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_write_buffer_number: 2 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compression: NoCompression 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bottommost_compression: Disabled 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.prefix_extractor: nullptr 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.num_levels: 7 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-13T14:47:52.691 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compression_opts.level: 32767 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compression_opts.strategy: 0 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-13T14:47:52.692 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compression_opts.enabled: false 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.target_file_size_base: 67108864 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-13T14:47:52.693 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.arena_block_size: 1048576 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.disable_auto_compactions: 0 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-13T14:47:52.694 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.table_properties_collectors: 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.inplace_update_support: 0 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.bloom_locality: 0 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.max_successive_merges: 0 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-13T14:47:52.695 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.paranoid_file_checks: 0 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.force_consistency_checks: 1 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.report_bg_io_stats: 0 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.ttl: 2592000 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.enable_blob_files: false 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.min_blob_size: 0 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.blob_file_size: 268435456 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-13T14:47:52.696 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.blob_file_starting_level: 0 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 38b0fbc8-8211-478c-a52f-f0d1108edc09 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707835672376360, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707835672376768, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707835672, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "38b0fbc8-8211-478c-a52f-f0d1108edc09", "db_session_id": "WGD14P953EZQRKKO5N87", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707835672376857, "job": 1, "event": "recovery_finished"} 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ecb119c000 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: DB pointer 0x55ecb1186000 2024-02-13T14:47:52.697 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: ** DB Stats ** 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: ** Compaction Stats [default] ** 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-13T14:47:52.698 INFO:journalctl@ceph.mon.b.smithi158.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: ** Compaction Stats [default] ** 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-13T14:47:52.699 INFO:journalctl@ceph.mon.b.smithi158.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout: Block cache BinnedLRUCache@0x55ecb10ab090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: using public_addr v2:172.21.15.158:0/0 -> [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: starting mon.b rank -1 at public addrs [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] at bind addrs [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(???) e0 preinit fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:47:52.700 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).mds e1 new map 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout: e1 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout: legacy client fscid: -1 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout: 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout: No filesystems configured 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T14:47:52.701 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Added host smithi158 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Updating smithi158:/etc/ceph/ceph.conf 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Updating smithi158:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.702 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi158", "caps": []}]: dispatch 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi158", "caps": []}]': finished 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Deploying daemon agent.smithi158 on smithi158 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:52.703 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi164", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Deploying cephadm binary to smithi164 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.704 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Added host smithi164 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Updating smithi164:/etc/ceph/ceph.conf 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:52.705 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]': finished 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Deploying daemon agent.smithi164 on smithi164 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.706 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/185142370' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/185142370' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-13T14:47:52.707 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi103:172.21.15.103=a;smithi158:172.21.15.158=b;smithi164:172.21.15.164=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Saving service mon spec with placement smithi103:172.21.15.103=a;smithi158:172.21.15.158=b;smithi164:172.21.15.164=c;count:3 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Deploying daemon mon.c on smithi164 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.708 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.709 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='client.? 172.21.15.164:0/944558291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T14:47:52.709 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:47:52.709 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.a calling monitor election 2024-02-13T14:47:52.709 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:52.709 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.709 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:52.709 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:52.709 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.c calling monitor election 2024-02-13T14:47:52.709 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.709 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: monmap e2: 2 mons at {a=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: fsmap 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mgrmap e13: y(active, since 52s) 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: overall HEALTH_OK 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.710 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: Deploying daemon mon.b on smithi158 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: from='client.? 172.21.15.164:0/2046542432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 ceph-mon[27628]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 bash[27591]: cd1f74efc812685ac576d27bf6410246a64c1421ee9aa0df6d6f26119b587f02 2024-02-13T14:47:52.711 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:52 smithi158 systemd[1]: Started Ceph mon.b for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:47:57.863 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:47:57.864 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: mon.c calling monitor election 2024-02-13T14:47:57.864 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: mon.a calling monitor election 2024-02-13T14:47:57.864 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.864 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:57.864 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.864 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:57.864 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.864 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.864 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:57.864 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.864 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: monmap e3: 3 mons at {a=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: fsmap 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: mgrmap e13: y(active, since 61s) 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: mon.b (rank 2) addr [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] is down (out of quorum) 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:57.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:57.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:47:57.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: mon.c calling monitor election 2024-02-13T14:47:57.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: mon.a calling monitor election 2024-02-13T14:47:57.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: monmap e3: 3 mons at {a=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: fsmap 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: mgrmap e13: y(active, since 61s) 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-13T14:47:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: mon.b (rank 2) addr [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] is down (out of quorum) 2024-02-13T14:47:57.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:57.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:57.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:57.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:57.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.319 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-02-13T14:47:58.320 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":3,"fsid":"7acdb230-ca7e-11ee-95b9-87774f69a715","modified":"2024-02-13T14:47:52.399672Z","created":"2024-02-13T14:45:50.651665Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:3300","nonce":0},{"type":"v1","addr":"172.21.15.158:6789","nonce":0}]},"addr":"172.21.15.158:6789/0","public_addr":"172.21.15.158:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-13T14:47:58.321 INFO:teuthology.orchestra.run.smithi164.stderr:dumped monmap epoch 3 2024-02-13T14:47:58.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:58.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:58.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: Updating smithi103:/etc/ceph/ceph.conf 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: Updating smithi158:/etc/ceph/ceph.conf 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: Updating smithi164:/etc/ceph/ceph.conf 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: Updating smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:47:58.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='client.? 172.21.15.164:0/3112189840' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T14:47:58.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:58.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:47:58.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T14:47:58.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:58.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: Updating smithi103:/etc/ceph/ceph.conf 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: Updating smithi158:/etc/ceph/ceph.conf 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: Updating smithi164:/etc/ceph/ceph.conf 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: Updating smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.907 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.907 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.907 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.907 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:58.907 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:47:58.907 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='client.? 172.21.15.164:0/3112189840' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T14:47:58.907 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:58.907 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:47:58.907 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T14:47:58.907 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:59.124 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-13T14:47:59.124 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph config generate-minimal-conf 2024-02-13T14:47:59.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:47:59.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: mon.c calling monitor election 2024-02-13T14:47:59.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: mon.a calling monitor election 2024-02-13T14:47:59.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:59.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:47:59.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:59.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:59.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:59.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:59.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: monmap e3: 3 mons at {a=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: fsmap 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: mgrmap e13: y(active, since 61s) 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: mon.b (rank 2) addr [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] is down (out of quorum) 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.680 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: Updating smithi103:/etc/ceph/ceph.conf 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: Updating smithi158:/etc/ceph/ceph.conf 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: Updating smithi164:/etc/ceph/ceph.conf 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: Updating smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:59.681 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='client.? 172.21.15.164:0/3112189840' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T14:47:59.682 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:00.038 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:48:00.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: mon.b calling monitor election 2024-02-13T14:48:00.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: mon.b calling monitor election 2024-02-13T14:48:00.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: mon.c calling monitor election 2024-02-13T14:48:00.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: mon.a calling monitor election 2024-02-13T14:48:00.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T14:48:00.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: monmap e3: 3 mons at {a=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T14:48:00.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: fsmap 2024-02-13T14:48:00.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:48:00.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: mgrmap e13: y(active, since 63s) 2024-02-13T14:48:00.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-13T14:48:00.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: Cluster is now healthy 2024-02-13T14:48:00.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:47:59 smithi164 ceph-mon[27636]: overall HEALTH_OK 2024-02-13T14:48:00.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: mon.b calling monitor election 2024-02-13T14:48:00.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: mon.b calling monitor election 2024-02-13T14:48:00.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: mon.c calling monitor election 2024-02-13T14:48:00.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: mon.a calling monitor election 2024-02-13T14:48:00.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T14:48:00.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: monmap e3: 3 mons at {a=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T14:48:00.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: fsmap 2024-02-13T14:48:00.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:48:00.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: mgrmap e13: y(active, since 63s) 2024-02-13T14:48:00.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-13T14:48:00.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: Cluster is now healthy 2024-02-13T14:48:00.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:47:59 smithi103 ceph-mon[25851]: overall HEALTH_OK 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: mon.b calling monitor election 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: mon.b calling monitor election 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: mon.c calling monitor election 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: mon.a calling monitor election 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: monmap e3: 3 mons at {a=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],b=[v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0],c=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: fsmap 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: osdmap e4: 0 total, 0 up, 0 in 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: mgrmap e13: y(active, since 63s) 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: Cluster is now healthy 2024-02-13T14:48:00.292 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:47:59 smithi158 ceph-mon[27628]: overall HEALTH_OK 2024-02-13T14:48:01.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: Reconfiguring mon.b (monmap changed)... 2024-02-13T14:48:01.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:48:01.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T14:48:01.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:01.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: Reconfiguring daemon mon.b on smithi158 2024-02-13T14:48:01.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:01.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:48:01.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: Reconfiguring mon.b (monmap changed)... 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: Reconfiguring daemon mon.b on smithi158 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.156 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.157 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.157 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.157 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.157 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.157 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: Reconfiguring mon.b (monmap changed)... 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: Reconfiguring daemon mon.b on smithi158 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.392 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.392 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.392 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.392 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.392 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:01.805 INFO:teuthology.orchestra.run.smithi103.stdout:# minimal ceph.conf for 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:48:01.805 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-02-13T14:48:01.805 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:48:01.805 INFO:teuthology.orchestra.run.smithi103.stdout: mon_host = [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-02-13T14:48:02.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:02 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:02.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:02 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/538782397' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:02.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:02 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:02.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:02 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:02.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:02 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:48:02.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:02 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T14:48:02.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:02 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:02.474 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-13T14:48:02.474 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:48:02.474 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.conf 2024-02-13T14:48:02.505 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:48:02.505 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:48:02.541 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:02 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:02.541 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:02 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/538782397' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:02.541 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:02 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:02.542 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:02 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:02.542 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:02 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:48:02.542 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:02 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T14:48:02.542 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:02 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:02.574 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-13T14:48:02.574 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/ceph/ceph.conf 2024-02-13T14:48:02.601 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:02 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:02.601 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:02 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/538782397' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:02.601 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:02 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:02.601 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:02 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:02.601 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:02 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-13T14:48:02.601 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:02 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-13T14:48:02.601 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:02 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:02.605 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-13T14:48:02.606 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:48:02.674 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-02-13T14:48:02.674 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.conf 2024-02-13T14:48:02.705 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-02-13T14:48:02.705 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:48:02.774 INFO:tasks.cephadm:Adding mgr.y on smithi103 2024-02-13T14:48:02.774 INFO:tasks.cephadm:Adding mgr.x on smithi158 2024-02-13T14:48:02.774 INFO:tasks.cephadm:Adding mgr.z on smithi164 2024-02-13T14:48:02.775 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch apply mgr '3;smithi103=y;smithi158=x;smithi164=z' 2024-02-13T14:48:03.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:03 smithi164 ceph-mon[27636]: Reconfiguring mon.c (monmap changed)... 2024-02-13T14:48:03.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:03 smithi164 ceph-mon[27636]: Reconfiguring daemon mon.c on smithi164 2024-02-13T14:48:03.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:03 smithi164 ceph-mon[27636]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:03.494 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.c/config 2024-02-13T14:48:03.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:03 smithi103 ceph-mon[25851]: Reconfiguring mon.c (monmap changed)... 2024-02-13T14:48:03.666 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:03 smithi103 ceph-mon[25851]: Reconfiguring daemon mon.c on smithi164 2024-02-13T14:48:03.666 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:03 smithi103 ceph-mon[25851]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:03.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:03 smithi158 ceph-mon[27628]: Reconfiguring mon.c (monmap changed)... 2024-02-13T14:48:03.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:03 smithi158 ceph-mon[27628]: Reconfiguring daemon mon.c on smithi164 2024-02-13T14:48:03.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:03 smithi158 ceph-mon[27628]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:04.536 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:04 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:04.536 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:04 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:04.536 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:04 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:04.536 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:04 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:04.536 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:04 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:04.536 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:04 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:04.537 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:04 smithi164 ceph-mon[27636]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:04.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:04 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:04.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:04 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:04.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:04 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:04.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:04 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:04.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:04 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:04.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:04 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:04.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:04 smithi103 ceph-mon[25851]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:04.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:04 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:04.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:04 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:04.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:04 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:04.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:04 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:04.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:04 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:04.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:04 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:04.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:04 smithi158 ceph-mon[27628]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:05.237 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled mgr update... 2024-02-13T14:48:05.817 DEBUG:teuthology.orchestra.run.smithi158:mgr.x> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.x.service 2024-02-13T14:48:05.821 DEBUG:teuthology.orchestra.run.smithi164:mgr.z> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.z.service 2024-02-13T14:48:05.823 INFO:tasks.cephadm:Deploying OSDs... 2024-02-13T14:48:05.823 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:48:05.823 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T14:48:05.842 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-13T14:48:05.842 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/nvme1n1 2024-02-13T14:48:05.901 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/nvme1n1 2024-02-13T14:48:05.901 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-02-13T14:48:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-13 14:47:49.697504320 +0000 2024-02-13T14:48:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-13 14:45:02.964943291 +0000 2024-02-13T14:48:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-13 14:45:02.964943291 +0000 2024-02-13T14:48:05.901 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-13 14:45:02.953943500 +0000 2024-02-13T14:48:05.901 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-02-13T14:48:05.971 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-13T14:48:05.971 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-13T14:48:05.971 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000226573 s, 2.3 MB/s 2024-02-13T14:48:05.972 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-02-13T14:48:06.030 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/nvme2n1 2024-02-13T14:48:06.086 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/nvme2n1 2024-02-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-02-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-13 14:47:49.708504112 +0000 2024-02-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-13 14:45:03.102940671 +0000 2024-02-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-13 14:45:03.102940671 +0000 2024-02-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-13 14:45:03.097940766 +0000 2024-02-13T14:48:06.087 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-02-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000299285 s, 1.7 MB/s 2024-02-13T14:48:06.155 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-02-13T14:48:06.212 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/nvme3n1 2024-02-13T14:48:06.268 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/nvme3n1 2024-02-13T14:48:06.268 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:06.269 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-02-13T14:48:06.269 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:06.269 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:06.269 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-13 14:47:49.719503904 +0000 2024-02-13T14:48:06.269 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-13 14:45:03.268937518 +0000 2024-02-13T14:48:06.269 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-13 14:45:03.268937518 +0000 2024-02-13T14:48:06.269 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-13 14:45:03.259937689 +0000 2024-02-13T14:48:06.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-02-13T14:48:06.336 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-13T14:48:06.336 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-13T14:48:06.336 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000246836 s, 2.1 MB/s 2024-02-13T14:48:06.338 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-02-13T14:48:06.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi103=y;smithi158=x;smithi164=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:48:06.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: Saving service mgr spec with placement smithi103=y;smithi158=x;smithi164=z;count:3 2024-02-13T14:48:06.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:06.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:06.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:06.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:06.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:06.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T14:48:06.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T14:48:06.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T14:48:06.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:06.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:06 smithi164 ceph-mon[27636]: Deploying daemon mgr.z on smithi164 2024-02-13T14:48:06.394 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/nvme4n1 2024-02-13T14:48:06.449 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/nvme4n1 2024-02-13T14:48:06.449 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:06.450 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-02-13T14:48:06.450 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:06.450 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:06.450 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-13 14:47:49.730503696 +0000 2024-02-13T14:48:06.450 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-13 14:45:03.377935448 +0000 2024-02-13T14:48:06.450 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-13 14:45:03.377935448 +0000 2024-02-13T14:48:06.450 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-13 14:45:03.365935676 +0000 2024-02-13T14:48:06.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-02-13T14:48:06.513 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi103=y;smithi158=x;smithi164=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:48:06.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: Saving service mgr spec with placement smithi103=y;smithi158=x;smithi164=z;count:3 2024-02-13T14:48:06.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:06.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:06.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:06.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:06.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:06.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T14:48:06.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T14:48:06.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T14:48:06.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:06.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:06 smithi103 ceph-mon[25851]: Deploying daemon mgr.z on smithi164 2024-02-13T14:48:06.516 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-13T14:48:06.516 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-13T14:48:06.516 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000243645 s, 2.1 MB/s 2024-02-13T14:48:06.517 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-02-13T14:48:06.573 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-13T14:48:06.573 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T14:48:06.592 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-13T14:48:06.593 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/nvme1n1 2024-02-13T14:48:06.650 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/nvme1n1 2024-02-13T14:48:06.651 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:06.651 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-02-13T14:48:06.651 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:06.651 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:06.651 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-02-13 14:48:00.040537251 +0000 2024-02-13T14:48:06.651 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-02-13 14:45:04.586501129 +0000 2024-02-13T14:48:06.651 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-02-13 14:45:04.586501129 +0000 2024-02-13T14:48:06.651 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-02-13 14:45:04.582501792 +0000 2024-02-13T14:48:06.651 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-02-13T14:48:06.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi103=y;smithi158=x;smithi164=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:48:06.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: Saving service mgr spec with placement smithi103=y;smithi158=x;smithi164=z;count:3 2024-02-13T14:48:06.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:06.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:06.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:06.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:06.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:06.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T14:48:06.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T14:48:06.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T14:48:06.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:06.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:06 smithi158 ceph-mon[27628]: Deploying daemon mgr.z on smithi164 2024-02-13T14:48:06.708 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-02-13T14:48:06.708 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-02-13T14:48:06.708 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000131088 s, 3.9 MB/s 2024-02-13T14:48:06.710 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-02-13T14:48:06.767 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/nvme2n1 2024-02-13T14:48:06.825 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/nvme2n1 2024-02-13T14:48:06.825 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:06.825 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-02-13T14:48:06.825 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:06.825 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:06.825 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-02-13 14:48:00.050537144 +0000 2024-02-13T14:48:06.825 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-02-13 14:45:04.740475594 +0000 2024-02-13T14:48:06.825 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-02-13 14:45:04.740475594 +0000 2024-02-13T14:48:06.825 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-02-13 14:45:04.730477252 +0000 2024-02-13T14:48:06.826 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-02-13T14:48:06.893 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-02-13T14:48:06.893 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-02-13T14:48:06.893 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000121035 s, 4.2 MB/s 2024-02-13T14:48:06.894 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-02-13T14:48:06.952 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/nvme3n1 2024-02-13T14:48:07.008 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/nvme3n1 2024-02-13T14:48:07.008 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:07.008 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-02-13T14:48:07.008 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:07.008 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:07.008 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-02-13 14:48:00.059537047 +0000 2024-02-13T14:48:07.008 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-02-13 14:45:04.886451385 +0000 2024-02-13T14:48:07.008 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-02-13 14:45:04.886451385 +0000 2024-02-13T14:48:07.008 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-02-13 14:45:04.878452712 +0000 2024-02-13T14:48:07.009 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-02-13T14:48:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-02-13T14:48:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-02-13T14:48:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000130756 s, 3.9 MB/s 2024-02-13T14:48:07.076 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-02-13T14:48:07.133 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/nvme4n1 2024-02-13T14:48:07.189 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/nvme4n1 2024-02-13T14:48:07.189 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:07.189 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-02-13T14:48:07.189 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:07.189 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:07.189 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2024-02-13 14:48:00.069536940 +0000 2024-02-13T14:48:07.189 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2024-02-13 14:45:05.035432195 +0000 2024-02-13T14:48:07.189 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2024-02-13 14:45:05.035432195 +0000 2024-02-13T14:48:07.189 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: 2024-02-13 14:45:05.024432314 +0000 2024-02-13T14:48:07.189 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-02-13T14:48:07.254 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2024-02-13T14:48:07.254 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2024-02-13T14:48:07.254 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000130341 s, 3.9 MB/s 2024-02-13T14:48:07.256 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-02-13T14:48:07.312 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-02-13T14:48:07.312 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-02-13T14:48:07.329 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-13T14:48:07.329 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/nvme1n1 2024-02-13T14:48:07.385 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/nvme1n1 2024-02-13T14:48:07.386 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:07.386 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-02-13T14:48:07.386 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:07.386 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:07.386 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-02-13 14:48:00.232380959 +0000 2024-02-13T14:48:07.386 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-02-13 14:45:06.256258981 +0000 2024-02-13T14:48:07.386 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-02-13 14:45:06.256258981 +0000 2024-02-13T14:48:07.386 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-02-13 14:45:06.244259180 +0000 2024-02-13T14:48:07.386 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-02-13T14:48:07.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:07 smithi164 ceph-mon[27636]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:07.417 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-02-13T14:48:07.417 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-02-13T14:48:07.417 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000265053 s, 1.9 MB/s 2024-02-13T14:48:07.419 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-02-13T14:48:07.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:07 smithi158 ceph-mon[27628]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:07.475 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/nvme2n1 2024-02-13T14:48:07.504 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:07 smithi103 ceph-mon[25851]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:07.529 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/nvme2n1 2024-02-13T14:48:07.529 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:07.530 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-02-13T14:48:07.530 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:07.530 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:07.530 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-02-13 14:48:00.243380778 +0000 2024-02-13T14:48:07.530 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-02-13 14:45:06.408256461 +0000 2024-02-13T14:48:07.530 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-02-13 14:45:06.408256461 +0000 2024-02-13T14:48:07.530 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-02-13 14:45:06.397256644 +0000 2024-02-13T14:48:07.530 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-02-13T14:48:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-02-13T14:48:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-02-13T14:48:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000242198 s, 2.1 MB/s 2024-02-13T14:48:07.598 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-02-13T14:48:07.654 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/nvme3n1 2024-02-13T14:48:07.710 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/nvme3n1 2024-02-13T14:48:07.710 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:07.710 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-02-13T14:48:07.710 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:07.710 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:07.710 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-02-13 14:48:00.254380596 +0000 2024-02-13T14:48:07.710 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-02-13 14:45:06.560253941 +0000 2024-02-13T14:48:07.710 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-02-13 14:45:06.560253941 +0000 2024-02-13T14:48:07.710 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-02-13 14:45:06.543254223 +0000 2024-02-13T14:48:07.711 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-02-13T14:48:07.778 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-02-13T14:48:07.778 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-02-13T14:48:07.778 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000254025 s, 2.0 MB/s 2024-02-13T14:48:07.779 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-02-13T14:48:07.835 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/nvme4n1 2024-02-13T14:48:07.892 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/nvme4n1 2024-02-13T14:48:07.892 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-13T14:48:07.892 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-02-13T14:48:07.892 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-13T14:48:07.892 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-13T14:48:07.892 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-02-13 14:48:00.265380415 +0000 2024-02-13T14:48:07.893 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-02-13 14:45:06.708251488 +0000 2024-02-13T14:48:07.893 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-02-13 14:45:06.708251488 +0000 2024-02-13T14:48:07.893 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-02-13 14:45:06.699251637 +0000 2024-02-13T14:48:07.893 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-02-13T14:48:07.958 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-02-13T14:48:07.958 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-02-13T14:48:07.959 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000203451 s, 2.5 MB/s 2024-02-13T14:48:07.960 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-02-13T14:48:08.016 INFO:tasks.cephadm:Deploying osd.0 on smithi103 with /dev/nvme4n1... 2024-02-13T14:48:08.017 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-02-13T14:48:08.265 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:48:08.640 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:08.411+0000 7f0f712a8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T14:48:09.131 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.131 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.131 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.132 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T14:48:09.132 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T14:48:09.132 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T14:48:09.132 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:09.132 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: Deploying daemon mgr.x on smithi158 2024-02-13T14:48:09.132 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:09.132 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.132 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.132 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:08 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T14:48:09.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T14:48:09.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T14:48:09.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:09.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: Deploying daemon mgr.x on smithi158 2024-02-13T14:48:09.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:09.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:08 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.284 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.284 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.284 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.284 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T14:48:09.284 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-13T14:48:09.284 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T14:48:09.284 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:09.284 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: Deploying daemon mgr.x on smithi158 2024-02-13T14:48:09.284 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:09.284 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.285 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.285 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:08 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:09.285 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:09.048+0000 7f0f712a8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T14:48:09.285 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:09.140+0000 7f0f712a8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T14:48:09.285 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:09.281+0000 7f0f712a8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T14:48:09.640 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:09.376+0000 7f0f712a8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T14:48:09.640 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:09.458+0000 7f0f712a8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T14:48:10.011 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:09.720+0000 7f0f712a8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T14:48:10.012 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:09.828+0000 7f0f712a8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T14:48:10.012 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:09.914+0000 7f0f712a8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T14:48:10.012 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:10 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:10.009+0000 7f0f712a8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T14:48:10.090 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:09 smithi158 systemd[1]: Starting Ceph mgr.x for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:48:10.390 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:10 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:10.185+0000 7f0f712a8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T14:48:10.400 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:10 smithi158 podman[29309]: 2024-02-13 14:48:10.087969986 +0000 UTC m=+0.106400318 container create 1f155c69d49ca9ed60e0adcf25d3a041a41ed926c194500fd99f1f2f465b436d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T14:48:10.400 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:10 smithi158 podman[29309]: 2024-02-13 14:48:09.999071704 +0000 UTC m=+0.017502011 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f 2024-02-13T14:48:10.400 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:10 smithi158 podman[29309]: 2024-02-13 14:48:10.307006337 +0000 UTC m=+0.325436627 container init 1f155c69d49ca9ed60e0adcf25d3a041a41ed926c194500fd99f1f2f465b436d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T14:48:10.400 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:10 smithi158 podman[29309]: 2024-02-13 14:48:10.312292704 +0000 UTC m=+0.330722994 container start 1f155c69d49ca9ed60e0adcf25d3a041a41ed926c194500fd99f1f2f465b436d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T14:48:10.400 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:10 smithi158 ceph-mgr[29347]: -- 172.21.15.158:0/1878750023 <== mon.2 v2:172.21.15.158:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5605b7e4e1a0 con 0x5605b7e25000 2024-02-13T14:48:10.400 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:10 smithi158 bash[29309]: 1f155c69d49ca9ed60e0adcf25d3a041a41ed926c194500fd99f1f2f465b436d 2024-02-13T14:48:10.678 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:10 smithi158 systemd[1]: Started Ceph mgr.x for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:48:10.692 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/nvme4n1 2024-02-13T14:48:10.692 INFO:teuthology.orchestra.run.smithi103.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-13T14:48:10.692 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-02-13T14:48:10.692 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-02-13T14:48:10.692 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-02-13T14:48:10.692 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0150873 s, 695 MB/s 2024-02-13T14:48:10.692 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-02-13T14:48:11.178 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:10 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:10.888+0000 7f0b9aafd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T14:48:11.178 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:10 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:10.983+0000 7f0b9aafd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T14:48:11.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi103:/dev/nvme4n1 2024-02-13T14:48:11.390 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:11.188+0000 7f0f712a8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T14:48:11.390 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:11.319+0000 7f0f712a8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T14:48:11.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:11.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:11.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:11.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:11.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T14:48:11.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T14:48:11.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:11.443 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:11.443 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.443 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.443 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.443 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.443 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:11.443 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:11.444 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:11.444 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.444 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T14:48:11.444 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T14:48:11.444 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:11.571 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:11.571 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.571 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.571 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.571 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.571 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:11.572 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:11.572 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:11.572 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:11.572 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-13T14:48:11.572 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-13T14:48:11.572 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:11.713 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:11.657+0000 7f0b9aafd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T14:48:11.761 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:11.667+0000 7f0f712a8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T14:48:11.761 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:11.758+0000 7f0f712a8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T14:48:11.996 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:11.759+0000 7f0b9aafd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T14:48:11.997 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:11.899+0000 7f0b9aafd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T14:48:11.997 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:11 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:11.994+0000 7f0b9aafd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T14:48:12.023 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:48:12.068 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:11 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:11.970+0000 7f0f712a8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:48:12.301 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:12.087+0000 7f0b9aafd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T14:48:12.390 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:12.157+0000 7f0f712a8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T14:48:12.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:12 smithi164 ceph-mon[27636]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:48:12.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:12 smithi164 ceph-mon[27636]: Reconfiguring mgr.y (unknown last config time)... 2024-02-13T14:48:12.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:12 smithi164 ceph-mon[27636]: Reconfiguring daemon mgr.y on smithi103 2024-02-13T14:48:12.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:12 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:12.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:12 smithi103 ceph-mon[25851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:48:12.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:12 smithi103 ceph-mon[25851]: Reconfiguring mgr.y (unknown last config time)... 2024-02-13T14:48:12.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:12 smithi103 ceph-mon[25851]: Reconfiguring daemon mgr.y on smithi103 2024-02-13T14:48:12.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:12 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:12.561 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:12.359+0000 7f0b9aafd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T14:48:12.561 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:12.468+0000 7f0b9aafd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T14:48:12.561 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:12.559+0000 7f0b9aafd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T14:48:12.562 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:12 smithi158 ceph-mon[27628]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:48:12.562 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:12 smithi158 ceph-mon[27628]: Reconfiguring mgr.y (unknown last config time)... 2024-02-13T14:48:12.562 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:12 smithi158 ceph-mon[27628]: Reconfiguring daemon mgr.y on smithi103 2024-02-13T14:48:12.562 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:12 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:12.745 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:12.489+0000 7f0f712a8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:48:12.858 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:12.663+0000 7f0b9aafd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T14:48:12.859 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:12.856+0000 7f0b9aafd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T14:48:13.044 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:12.787+0000 7f0f712a8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T14:48:13.044 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:13 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:13.042+0000 7f0f712a8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T14:48:13.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:13 smithi158 ceph-mon[27628]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:13.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:13 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:13.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:13 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:13.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:13 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:13.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:13 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:13.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:13 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:13.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:13 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:13.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:13 smithi164 ceph-mon[27636]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:13.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:13 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:13.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:13 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:13.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:13 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:13.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:13 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:13.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:13 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:13.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:13 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:13.589 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:13 smithi103 ceph-mon[25851]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:13.589 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:13 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:13.589 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:13 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:13.589 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:13 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:13.589 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:13 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:13.589 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:13 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:13.589 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:13 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.140 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:13 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:13.857+0000 7f0f712a8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T14:48:14.178 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:13 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:13.898+0000 7f0b9aafd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T14:48:14.178 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:14 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:14.033+0000 7f0b9aafd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T14:48:14.640 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:14 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:14.443+0000 7f0f712a8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T14:48:14.640 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:14 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:14.541+0000 7f0f712a8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T14:48:14.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:14 smithi164 ceph-mon[27636]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:48:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:14 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:14 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:14 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:14 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:14 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:14 smithi103 ceph-mon[25851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:48:14.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:14 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:14 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:14 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:14 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:14 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.668 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:14 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:14.381+0000 7f0b9aafd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T14:48:14.668 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:14 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:14.473+0000 7f0b9aafd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T14:48:14.668 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:14 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:14.666+0000 7f0b9aafd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:48:14.668 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:14 smithi158 ceph-mon[27628]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:48:14.668 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:14 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.668 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:14 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.668 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:14 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.669 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:14 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.669 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:14 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:14.928 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:14 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:14.849+0000 7f0b9aafd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T14:48:15.140 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:48:14 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:48:14.753+0000 7f0f712a8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T14:48:15.396 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:15.131+0000 7f0b9aafd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:48:15.397 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:15.397 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.397 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.397 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.397 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:48:15.397 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:15.397 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T14:48:15.397 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T14:48:15.397 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:15.398 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:15.398 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:15.398 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.398 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: Standby manager daemon z started 2024-02-13T14:48:15.398 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-13T14:48:15.398 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:48:15.398 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-13T14:48:15.398 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:48:15.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:15.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: Standby manager daemon z started 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-13T14:48:15.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:15 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:48:15.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:15.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:48:15.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T14:48:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T14:48:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: Standby manager daemon z started 2024-02-13T14:48:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-13T14:48:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:48:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-13T14:48:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:15 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.164:0/2821703223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:48:15.678 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:15.394+0000 7f0b9aafd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T14:48:15.678 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:15 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:15.627+0000 7f0b9aafd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T14:48:16.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:16 smithi164 ceph-mon[27636]: mgrmap e14: y(active, since 79s), standbys: z 2024-02-13T14:48:16.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:16 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-13T14:48:16.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:16 smithi103 ceph-mon[25851]: mgrmap e14: y(active, since 79s), standbys: z 2024-02-13T14:48:16.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:16 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-13T14:48:16.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:16 smithi158 ceph-mon[27628]: mgrmap e14: y(active, since 79s), standbys: z 2024-02-13T14:48:16.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:16 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-13T14:48:16.678 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:16 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:16.426+0000 7f0b9aafd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T14:48:17.312 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:16.999+0000 7f0b9aafd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T14:48:17.312 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:17.095+0000 7f0b9aafd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T14:48:17.312 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:48:17.309+0000 7f0b9aafd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T14:48:17.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:17 smithi164 ceph-mon[27636]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:17.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:17 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/1606134289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2c5b1d3-1c27-44d0-87a6-45b988205060"}]: dispatch 2024-02-13T14:48:17.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:17 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/1606134289' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2c5b1d3-1c27-44d0-87a6-45b988205060"}]': finished 2024-02-13T14:48:17.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:17 smithi164 ceph-mon[27636]: osdmap e5: 1 total, 0 up, 1 in 2024-02-13T14:48:17.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:17 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:17.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:17 smithi164 ceph-mon[27636]: Standby manager daemon x started 2024-02-13T14:48:17.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:17 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T14:48:17.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:17 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:48:17.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:17 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T14:48:17.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:17 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:48:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:17 smithi103 ceph-mon[25851]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:17 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/1606134289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2c5b1d3-1c27-44d0-87a6-45b988205060"}]: dispatch 2024-02-13T14:48:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:17 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/1606134289' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2c5b1d3-1c27-44d0-87a6-45b988205060"}]': finished 2024-02-13T14:48:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:17 smithi103 ceph-mon[25851]: osdmap e5: 1 total, 0 up, 1 in 2024-02-13T14:48:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:17 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:17 smithi103 ceph-mon[25851]: Standby manager daemon x started 2024-02-13T14:48:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:17 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T14:48:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:17 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:48:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:17 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T14:48:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:17 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:48:17.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-mon[27628]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:17.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/1606134289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2c5b1d3-1c27-44d0-87a6-45b988205060"}]: dispatch 2024-02-13T14:48:17.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/1606134289' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2c5b1d3-1c27-44d0-87a6-45b988205060"}]': finished 2024-02-13T14:48:17.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-mon[27628]: osdmap e5: 1 total, 0 up, 1 in 2024-02-13T14:48:17.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:17.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-mon[27628]: Standby manager daemon x started 2024-02-13T14:48:17.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T14:48:17.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:48:17.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T14:48:17.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:17 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.158:0/1973986638' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:48:19.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:18 smithi158 ceph-mon[27628]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-02-13T14:48:19.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:18 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T14:48:19.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:18 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/1223438451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T14:48:19.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:18 smithi158 ceph-mon[27628]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:19.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:18 smithi164 ceph-mon[27636]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-02-13T14:48:19.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:18 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T14:48:19.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:18 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/1223438451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T14:48:19.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:18 smithi164 ceph-mon[27636]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:19.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:18 smithi103 ceph-mon[25851]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-02-13T14:48:19.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:18 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T14:48:19.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:18 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/1223438451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T14:48:19.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:18 smithi103 ceph-mon[25851]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:21.568 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:21 smithi103 ceph-mon[25851]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:21.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:21 smithi164 ceph-mon[27636]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:21.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:21 smithi158 ceph-mon[27628]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:23.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:23 smithi103 ceph-mon[25851]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:23.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:23 smithi164 ceph-mon[27636]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:23.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:23 smithi158 ceph-mon[27628]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:24.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:24 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T14:48:24.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:24 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:24.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:24 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T14:48:24.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:24 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:24.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:24 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-13T14:48:24.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:24 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:25.569 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:25 smithi103 ceph-mon[25851]: Deploying daemon osd.0 on smithi103 2024-02-13T14:48:25.569 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:25 smithi103 ceph-mon[25851]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:25.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:25 smithi164 ceph-mon[27636]: Deploying daemon osd.0 on smithi103 2024-02-13T14:48:25.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:25 smithi164 ceph-mon[27636]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:25.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:25 smithi158 ceph-mon[27628]: Deploying daemon osd.0 on smithi103 2024-02-13T14:48:25.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:25 smithi158 ceph-mon[27628]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:27.452 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:27 smithi103 ceph-mon[25851]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:27.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:27 smithi164 ceph-mon[27636]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:27.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:27 smithi158 ceph-mon[27628]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:29.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:29 smithi103 ceph-mon[25851]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:29.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:29 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:29.550 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:29 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:29.550 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:29 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:29.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:29 smithi164 ceph-mon[27636]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:29.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:29 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:29.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:29 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:29.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:29 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:29.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:29 smithi158 ceph-mon[27628]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:29.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:29 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:29.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:29 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:29.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:29 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:31 smithi103 ceph-mon[25851]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:31.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:31.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:31.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:31.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:31 smithi164 ceph-mon[27636]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:31.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:31.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:31.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:31.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:31 smithi158 ceph-mon[27628]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:31.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:31.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:31.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:31.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:31.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:33.596 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:33 smithi103 ceph-mon[25851]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:33.596 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:33.596 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:33.596 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:33.610 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:33 smithi164 ceph-mon[27636]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:33.611 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:33.611 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:33.611 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:33.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:33 smithi158 ceph-mon[27628]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:33.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:33.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:33.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:33.976 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 0 on host 'smithi103' 2024-02-13T14:48:34.524 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T14:48:34.524 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.524 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:34.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:48:34.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:34.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:34.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.551 DEBUG:teuthology.orchestra.run.smithi103:osd.0> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@osd.0.service 2024-02-13T14:48:34.553 INFO:tasks.cephadm:Deploying osd.1 on smithi158 with /dev/nvme4n1... 2024-02-13T14:48:34.553 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-02-13T14:48:34.580 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T14:48:34.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:34.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:48:34.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:34.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:34.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-13T14:48:34.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:34.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:48:34.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:34.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:34.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:34.794 INFO:teuthology.orchestra.run.smithi158.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T14:48:35.599 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:35 smithi158 ceph-mon[27628]: Adjusting osd_memory_target on smithi103 to 16064M 2024-02-13T14:48:35.599 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:35 smithi158 ceph-mon[27628]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:35.599 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:35 smithi158 ceph-mon[27628]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T14:48:35.599 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:35 smithi158 ceph-mon[27628]: osdmap e6: 1 total, 0 up, 1 in 2024-02-13T14:48:35.600 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:35 smithi158 ceph-mon[27628]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-13T14:48:35.600 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:35 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:35.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:35 smithi164 ceph-mon[27636]: Adjusting osd_memory_target on smithi103 to 16064M 2024-02-13T14:48:35.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:35 smithi164 ceph-mon[27636]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:35.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:35 smithi164 ceph-mon[27636]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T14:48:35.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:35 smithi164 ceph-mon[27636]: osdmap e6: 1 total, 0 up, 1 in 2024-02-13T14:48:35.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:35 smithi164 ceph-mon[27636]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-13T14:48:35.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:35 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:35.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:35 smithi103 ceph-mon[25851]: Adjusting osd_memory_target on smithi103 to 16064M 2024-02-13T14:48:35.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:35 smithi103 ceph-mon[25851]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:35.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:35 smithi103 ceph-mon[25851]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-13T14:48:35.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:35 smithi103 ceph-mon[25851]: osdmap e6: 1 total, 0 up, 1 in 2024-02-13T14:48:35.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:35 smithi103 ceph-mon[25851]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-13T14:48:35.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:35 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:35.655 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:48:35 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0[38276]: 2024-02-13T14:48:35.317+0000 7f444117b700 -1 osd.0 0 waiting for initial osdmap 2024-02-13T14:48:35.656 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:48:35 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0[38276]: 2024-02-13T14:48:35.321+0000 7f443cf6e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T14:48:36.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:36 smithi158 ceph-mon[27628]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-02-13T14:48:36.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:36 smithi158 ceph-mon[27628]: osdmap e7: 1 total, 0 up, 1 in 2024-02-13T14:48:36.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:36 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:36.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:36 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:36.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:36 smithi164 ceph-mon[27636]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-02-13T14:48:36.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:36 smithi164 ceph-mon[27636]: osdmap e7: 1 total, 0 up, 1 in 2024-02-13T14:48:36.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:36 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:36.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:36 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:36.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:36 smithi103 ceph-mon[25851]: from='osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-02-13T14:48:36.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:36 smithi103 ceph-mon[25851]: osdmap e7: 1 total, 0 up, 1 in 2024-02-13T14:48:36.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:36 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:36.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:36 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:36.684 INFO:teuthology.orchestra.run.smithi158.stderr:--> Zapping: /dev/nvme4n1 2024-02-13T14:48:36.684 INFO:teuthology.orchestra.run.smithi158.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-13T14:48:36.684 INFO:teuthology.orchestra.run.smithi158.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-02-13T14:48:36.684 INFO:teuthology.orchestra.run.smithi158.stderr: stderr: 10+0 records in 2024-02-13T14:48:36.684 INFO:teuthology.orchestra.run.smithi158.stderr:10+0 records out 2024-02-13T14:48:36.684 INFO:teuthology.orchestra.run.smithi158.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00989089 s, 1.1 GB/s 2024-02-13T14:48:36.684 INFO:teuthology.orchestra.run.smithi158.stderr:--> Zapping successful for: 2024-02-13T14:48:37.301 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi158:/dev/nvme4n1 2024-02-13T14:48:37.581 INFO:teuthology.orchestra.run.smithi158.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T14:48:37.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:37 smithi164 ceph-mon[27636]: purged_snaps scrub starts 2024-02-13T14:48:37.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:37 smithi164 ceph-mon[27636]: purged_snaps scrub ok 2024-02-13T14:48:37.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:37 smithi164 ceph-mon[27636]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:37.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:37 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:37.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:37 smithi164 ceph-mon[27636]: osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222] boot 2024-02-13T14:48:37.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:37 smithi164 ceph-mon[27636]: osdmap e8: 1 total, 1 up, 1 in 2024-02-13T14:48:37.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:37 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:37.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:37 smithi103 ceph-mon[25851]: purged_snaps scrub starts 2024-02-13T14:48:37.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:37 smithi103 ceph-mon[25851]: purged_snaps scrub ok 2024-02-13T14:48:37.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:37 smithi103 ceph-mon[25851]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:37.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:37 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:37.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:37 smithi103 ceph-mon[25851]: osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222] boot 2024-02-13T14:48:37.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:37 smithi103 ceph-mon[25851]: osdmap e8: 1 total, 1 up, 1 in 2024-02-13T14:48:37.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:37 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:37 smithi158 ceph-mon[27628]: purged_snaps scrub starts 2024-02-13T14:48:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:37 smithi158 ceph-mon[27628]: purged_snaps scrub ok 2024-02-13T14:48:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:37 smithi158 ceph-mon[27628]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-13T14:48:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:37 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:37 smithi158 ceph-mon[27628]: osd.0 [v2:172.21.15.103:6802/2605990222,v1:172.21.15.103:6803/2605990222] boot 2024-02-13T14:48:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:37 smithi158 ceph-mon[27628]: osdmap e8: 1 total, 1 up, 1 in 2024-02-13T14:48:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:37 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:48:38.503 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:38 smithi158 ceph-mon[27628]: osdmap e9: 1 total, 1 up, 1 in 2024-02-13T14:48:38.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:38 smithi164 ceph-mon[27636]: osdmap e9: 1 total, 1 up, 1 in 2024-02-13T14:48:38.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:38 smithi103 ceph-mon[25851]: osdmap e9: 1 total, 1 up, 1 in 2024-02-13T14:48:39.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:39 smithi164 ceph-mon[27636]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:39.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:39 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T14:48:39.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:39 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T14:48:39.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:39 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:39.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:39 smithi103 ceph-mon[25851]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:39.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:39 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T14:48:39.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:39 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T14:48:39.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:39 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:39.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:39 smithi158 ceph-mon[27628]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:39.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:39 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T14:48:39.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:39 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T14:48:39.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:39 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:40.403 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:40 smithi158 ceph-mon[27628]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:48:40.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:40 smithi164 ceph-mon[27636]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:48:40.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:40 smithi103 ceph-mon[25851]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi158:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:48:41.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:41 smithi164 ceph-mon[27636]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:41.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:41 smithi103 ceph-mon[25851]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:41.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:41 smithi158 ceph-mon[27628]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:42.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:42 smithi164 ceph-mon[27636]: from='client.? 172.21.15.158:0/3801798304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bd2fdeb-bc00-434a-9e38-831a4e8be19f"}]: dispatch 2024-02-13T14:48:42.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:42 smithi164 ceph-mon[27636]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bd2fdeb-bc00-434a-9e38-831a4e8be19f"}]: dispatch 2024-02-13T14:48:42.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:42 smithi164 ceph-mon[27636]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bd2fdeb-bc00-434a-9e38-831a4e8be19f"}]': finished 2024-02-13T14:48:42.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:42 smithi164 ceph-mon[27636]: osdmap e10: 2 total, 1 up, 2 in 2024-02-13T14:48:42.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:42 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:48:42.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:42 smithi103 ceph-mon[25851]: from='client.? 172.21.15.158:0/3801798304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bd2fdeb-bc00-434a-9e38-831a4e8be19f"}]: dispatch 2024-02-13T14:48:42.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:42 smithi103 ceph-mon[25851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bd2fdeb-bc00-434a-9e38-831a4e8be19f"}]: dispatch 2024-02-13T14:48:42.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:42 smithi103 ceph-mon[25851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bd2fdeb-bc00-434a-9e38-831a4e8be19f"}]': finished 2024-02-13T14:48:42.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:42 smithi103 ceph-mon[25851]: osdmap e10: 2 total, 1 up, 2 in 2024-02-13T14:48:42.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:42 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:48:42.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:42 smithi158 ceph-mon[27628]: from='client.? 172.21.15.158:0/3801798304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bd2fdeb-bc00-434a-9e38-831a4e8be19f"}]: dispatch 2024-02-13T14:48:42.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:42 smithi158 ceph-mon[27628]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bd2fdeb-bc00-434a-9e38-831a4e8be19f"}]: dispatch 2024-02-13T14:48:42.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:42 smithi158 ceph-mon[27628]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bd2fdeb-bc00-434a-9e38-831a4e8be19f"}]': finished 2024-02-13T14:48:42.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:42 smithi158 ceph-mon[27628]: osdmap e10: 2 total, 1 up, 2 in 2024-02-13T14:48:42.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:42 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:48:43.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:43 smithi164 ceph-mon[27636]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:43.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:43 smithi164 ceph-mon[27636]: from='client.? 172.21.15.158:0/2430555727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T14:48:43.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:43 smithi103 ceph-mon[25851]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:43.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:43 smithi103 ceph-mon[25851]: from='client.? 172.21.15.158:0/2430555727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T14:48:43.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:43 smithi158 ceph-mon[27628]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:43.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:43 smithi158 ceph-mon[27628]: from='client.? 172.21.15.158:0/2430555727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T14:48:45.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:45 smithi164 ceph-mon[27636]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:45.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:45 smithi103 ceph-mon[25851]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:45.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:45 smithi158 ceph-mon[27628]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:47.445 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:47 smithi158 ceph-mon[27628]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:47.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:47 smithi164 ceph-mon[27636]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:47.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:47 smithi103 ceph-mon[25851]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:49.493 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:49 smithi103 ceph-mon[25851]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:49.493 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:49 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T14:48:49.493 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:49 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:49.587 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:49 smithi158 ceph-mon[27628]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:49.587 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:49 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T14:48:49.587 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:49 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:49.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:49 smithi164 ceph-mon[27636]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:49.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:49 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-13T14:48:49.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:49 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:50.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:50 smithi164 ceph-mon[27636]: Deploying daemon osd.1 on smithi158 2024-02-13T14:48:50.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:50 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:50.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:50 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:50.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:50 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:50.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:50 smithi158 ceph-mon[27628]: Deploying daemon osd.1 on smithi158 2024-02-13T14:48:50.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:50 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:50.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:50 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:50.641 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:50 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:50.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:50 smithi103 ceph-mon[25851]: Deploying daemon osd.1 on smithi158 2024-02-13T14:48:50.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:50 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:50.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:50 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:50.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:50 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:51.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:51 smithi103 ceph-mon[25851]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:51.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:51 smithi103 ceph-mon[25851]: Detected new or changed devices on smithi103 2024-02-13T14:48:51.637 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:51 smithi158 ceph-mon[27628]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:51.637 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:51 smithi158 ceph-mon[27628]: Detected new or changed devices on smithi103 2024-02-13T14:48:51.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:51 smithi164 ceph-mon[27636]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:51.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:51 smithi164 ceph-mon[27636]: Detected new or changed devices on smithi103 2024-02-13T14:48:52.476 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:52 smithi158 ceph-mon[27628]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:52.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:52 smithi164 ceph-mon[27636]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:52.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:52 smithi103 ceph-mon[25851]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:53.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:53 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:53.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:53 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:53.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:53 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:53.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:53 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:53.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:53 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:53.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:53 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:53 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:53 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:53 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:53 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:53 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:53 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:53 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:53 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:53 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:53 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:53 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:53 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.853 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:54 smithi158 ceph-mon[27628]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:54.880 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:54 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.881 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:54 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.881 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:54 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:54.881 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:54 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:54.881 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:54 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:54.881 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:54 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:54 smithi164 ceph-mon[27636]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:54.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:54 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:54 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:54.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:54 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:54.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:54 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:54.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:54 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:54.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:54 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:55.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:54 smithi103 ceph-mon[25851]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:55.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:54 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:55.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:54 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:55.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:54 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:55.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:54 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:55.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:54 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:55.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:54 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:57.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:57 smithi158 ceph-mon[27628]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:57.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:57 smithi164 ceph-mon[27636]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:57.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:57 smithi103 ceph-mon[25851]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:58.314 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:58 smithi158 ceph-mon[27628]: from='osd.1 [v2:172.21.15.158:6800/2266577857,v1:172.21.15.158:6801/2266577857]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T14:48:58.315 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:58 smithi158 ceph-mon[27628]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T14:48:58.315 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:58 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:58.315 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:58 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:58.315 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:58 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:58.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:58 smithi103 ceph-mon[25851]: from='osd.1 [v2:172.21.15.158:6800/2266577857,v1:172.21.15.158:6801/2266577857]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T14:48:58.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:58 smithi103 ceph-mon[25851]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T14:48:58.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:58.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:58.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:58 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:58.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:58 smithi164 ceph-mon[27636]: from='osd.1 [v2:172.21.15.158:6800/2266577857,v1:172.21.15.158:6801/2266577857]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T14:48:58.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:58 smithi164 ceph-mon[27636]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-13T14:48:58.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:58.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:58.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:58 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.028 INFO:teuthology.orchestra.run.smithi158.stdout:Created osd(s) 1 on host 'smithi158' 2024-02-13T14:48:59.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: Detected new or changed devices on smithi158 2024-02-13T14:48:59.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:59.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T14:48:59.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='osd.1 [v2:172.21.15.158:6800/2266577857,v1:172.21.15.158:6801/2266577857]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2024-02-13T14:48:59.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: osdmap e11: 2 total, 1 up, 2 in 2024-02-13T14:48:59.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:48:59.581 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2024-02-13T14:48:59.582 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.582 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.582 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:59.582 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:48:59.582 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.582 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:59.582 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:59.582 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:48:59 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.613 DEBUG:teuthology.orchestra.run.smithi158:osd.1> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@osd.1.service 2024-02-13T14:48:59.616 INFO:tasks.cephadm:Deploying osd.2 on smithi164 with /dev/nvme4n1... 2024-02-13T14:48:59.616 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-02-13T14:48:59.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: Detected new or changed devices on smithi158 2024-02-13T14:48:59.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:59.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T14:48:59.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='osd.1 [v2:172.21.15.158:6800/2266577857,v1:172.21.15.158:6801/2266577857]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2024-02-13T14:48:59.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: osdmap e11: 2 total, 1 up, 2 in 2024-02-13T14:48:59.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:48:59.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2024-02-13T14:48:59.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:59.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:48:59.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:59.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:59.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:48:59 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: Detected new or changed devices on smithi158 2024-02-13T14:48:59.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:48:59.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-13T14:48:59.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='osd.1 [v2:172.21.15.158:6800/2266577857,v1:172.21.15.158:6801/2266577857]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2024-02-13T14:48:59.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: osdmap e11: 2 total, 1 up, 2 in 2024-02-13T14:48:59.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:48:59.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]: dispatch 2024-02-13T14:48:59.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:48:59.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:48:59.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:48:59.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:48:59.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:48:59 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:48:59.863 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.c/config 2024-02-13T14:49:00.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:00 smithi164 ceph-mon[27636]: Adjusting osd_memory_target on smithi158 to 16064M 2024-02-13T14:49:00.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:00 smithi164 ceph-mon[27636]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2024-02-13T14:49:00.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:00 smithi164 ceph-mon[27636]: osdmap e12: 2 total, 1 up, 2 in 2024-02-13T14:49:00.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:00 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:00.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:00 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:00.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:00 smithi164 ceph-mon[27636]: from='osd.1 ' entity='osd.1' 2024-02-13T14:49:00.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:00 smithi103 ceph-mon[25851]: Adjusting osd_memory_target on smithi158 to 16064M 2024-02-13T14:49:00.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:00 smithi103 ceph-mon[25851]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2024-02-13T14:49:00.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:00 smithi103 ceph-mon[25851]: osdmap e12: 2 total, 1 up, 2 in 2024-02-13T14:49:00.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:00 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:00.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:00 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:00.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:00 smithi103 ceph-mon[25851]: from='osd.1 ' entity='osd.1' 2024-02-13T14:49:00.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:00 smithi158 ceph-mon[27628]: Adjusting osd_memory_target on smithi158 to 16064M 2024-02-13T14:49:00.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:00 smithi158 ceph-mon[27628]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi158", "root=default"]}]': finished 2024-02-13T14:49:00.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:00 smithi158 ceph-mon[27628]: osdmap e12: 2 total, 1 up, 2 in 2024-02-13T14:49:00.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:00 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:00.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:00 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:00.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:00 smithi158 ceph-mon[27628]: from='osd.1 ' entity='osd.1' 2024-02-13T14:49:01.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:01 smithi164 ceph-mon[27636]: purged_snaps scrub starts 2024-02-13T14:49:01.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:01 smithi164 ceph-mon[27636]: purged_snaps scrub ok 2024-02-13T14:49:01.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:01 smithi164 ceph-mon[27636]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:49:01.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:01.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:01 smithi164 ceph-mon[27636]: osd.1 [v2:172.21.15.158:6800/2266577857,v1:172.21.15.158:6801/2266577857] boot 2024-02-13T14:49:01.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:01 smithi164 ceph-mon[27636]: osdmap e13: 2 total, 2 up, 2 in 2024-02-13T14:49:01.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:01 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:01.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:01 smithi103 ceph-mon[25851]: purged_snaps scrub starts 2024-02-13T14:49:01.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:01 smithi103 ceph-mon[25851]: purged_snaps scrub ok 2024-02-13T14:49:01.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:01 smithi103 ceph-mon[25851]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:49:01.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:01.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:01 smithi103 ceph-mon[25851]: osd.1 [v2:172.21.15.158:6800/2266577857,v1:172.21.15.158:6801/2266577857] boot 2024-02-13T14:49:01.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:01 smithi103 ceph-mon[25851]: osdmap e13: 2 total, 2 up, 2 in 2024-02-13T14:49:01.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:01 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:01.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:01 smithi158 ceph-mon[27628]: purged_snaps scrub starts 2024-02-13T14:49:01.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:01 smithi158 ceph-mon[27628]: purged_snaps scrub ok 2024-02-13T14:49:01.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:01 smithi158 ceph-mon[27628]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-13T14:49:01.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:01.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:01 smithi158 ceph-mon[27628]: osd.1 [v2:172.21.15.158:6800/2266577857,v1:172.21.15.158:6801/2266577857] boot 2024-02-13T14:49:01.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:01 smithi158 ceph-mon[27628]: osdmap e13: 2 total, 2 up, 2 in 2024-02-13T14:49:01.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:01 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:49:01.830 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/nvme4n1 2024-02-13T14:49:01.830 INFO:teuthology.orchestra.run.smithi164.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-13T14:49:01.830 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-02-13T14:49:01.830 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-02-13T14:49:01.831 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-02-13T14:49:01.831 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129755 s, 808 MB/s 2024-02-13T14:49:01.831 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-02-13T14:49:02.444 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi164:/dev/nvme4n1 2024-02-13T14:49:02.722 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.c/config 2024-02-13T14:49:02.834 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:02 smithi164 ceph-mon[27636]: osdmap e14: 2 total, 2 up, 2 in 2024-02-13T14:49:02.835 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:02 smithi164 ceph-mon[27636]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:02.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:02 smithi103 ceph-mon[25851]: osdmap e14: 2 total, 2 up, 2 in 2024-02-13T14:49:02.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:02 smithi103 ceph-mon[25851]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:02.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:02 smithi158 ceph-mon[27628]: osdmap e14: 2 total, 2 up, 2 in 2024-02-13T14:49:02.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:02 smithi158 ceph-mon[27628]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:05.309 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:05 smithi164 ceph-mon[27636]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:05.309 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:05 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T14:49:05.309 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:05 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T14:49:05.309 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:05 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:05.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:05 smithi103 ceph-mon[25851]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:05.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:05 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T14:49:05.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:05 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T14:49:05.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:05 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:05.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:05 smithi158 ceph-mon[27628]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:05.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:05 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-13T14:49:05.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:05 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-13T14:49:05.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:05 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:06.323 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:06 smithi164 ceph-mon[27636]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:06.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:06 smithi103 ceph-mon[25851]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:06.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:06 smithi158 ceph-mon[27628]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:07.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:07 smithi164 ceph-mon[27636]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:07.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:07 smithi164 ceph-mon[27636]: from='client.? 172.21.15.164:0/4133705515' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f338f379-8817-43f4-a81e-6a851fd7196b"}]: dispatch 2024-02-13T14:49:07.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:07 smithi164 ceph-mon[27636]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f338f379-8817-43f4-a81e-6a851fd7196b"}]: dispatch 2024-02-13T14:49:07.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:07 smithi164 ceph-mon[27636]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f338f379-8817-43f4-a81e-6a851fd7196b"}]': finished 2024-02-13T14:49:07.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:07 smithi164 ceph-mon[27636]: osdmap e15: 3 total, 2 up, 3 in 2024-02-13T14:49:07.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:07 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:07.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:07 smithi103 ceph-mon[25851]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:07.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:07 smithi103 ceph-mon[25851]: from='client.? 172.21.15.164:0/4133705515' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f338f379-8817-43f4-a81e-6a851fd7196b"}]: dispatch 2024-02-13T14:49:07.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:07 smithi103 ceph-mon[25851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f338f379-8817-43f4-a81e-6a851fd7196b"}]: dispatch 2024-02-13T14:49:07.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:07 smithi103 ceph-mon[25851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f338f379-8817-43f4-a81e-6a851fd7196b"}]': finished 2024-02-13T14:49:07.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:07 smithi103 ceph-mon[25851]: osdmap e15: 3 total, 2 up, 3 in 2024-02-13T14:49:07.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:07 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:07.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:07 smithi158 ceph-mon[27628]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:07.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:07 smithi158 ceph-mon[27628]: from='client.? 172.21.15.164:0/4133705515' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f338f379-8817-43f4-a81e-6a851fd7196b"}]: dispatch 2024-02-13T14:49:07.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:07 smithi158 ceph-mon[27628]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f338f379-8817-43f4-a81e-6a851fd7196b"}]: dispatch 2024-02-13T14:49:07.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:07 smithi158 ceph-mon[27628]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f338f379-8817-43f4-a81e-6a851fd7196b"}]': finished 2024-02-13T14:49:07.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:07 smithi158 ceph-mon[27628]: osdmap e15: 3 total, 2 up, 3 in 2024-02-13T14:49:07.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:07 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:09.540 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:09 smithi103 ceph-mon[25851]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:09.540 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:09 smithi103 ceph-mon[25851]: from='client.? 172.21.15.164:0/596686965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T14:49:09.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:09 smithi164 ceph-mon[27636]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:09.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:09 smithi164 ceph-mon[27636]: from='client.? 172.21.15.164:0/596686965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T14:49:09.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:09 smithi158 ceph-mon[27628]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:09.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:09 smithi158 ceph-mon[27628]: from='client.? 172.21.15.164:0/596686965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-13T14:49:11.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:11 smithi164 ceph-mon[27636]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:11.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:11.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:11.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:11 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:11.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:11 smithi103 ceph-mon[25851]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:11.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:11.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:11.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:11 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:11.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:11 smithi158 ceph-mon[27628]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:11.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:11.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:11.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:11 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:13.320 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:13 smithi164 ceph-mon[27636]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:13.320 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:13 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:13.320 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:13 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:13.320 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:13 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:13.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:13 smithi103 ceph-mon[25851]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:13.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:13 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:13.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:13 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:13.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:13 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:13.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:13 smithi158 ceph-mon[27628]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:13.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:13 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:13.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:13 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:13.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:13 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:15.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:15 smithi164 ceph-mon[27636]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:15.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T14:49:15.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:15 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:15.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:15 smithi103 ceph-mon[25851]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:15.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T14:49:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:15 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:15 smithi158 ceph-mon[27628]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-13T14:49:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:15 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:16.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:16 smithi164 ceph-mon[27636]: Deploying daemon osd.2 on smithi164 2024-02-13T14:49:16.526 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:16 smithi158 ceph-mon[27628]: Deploying daemon osd.2 on smithi164 2024-02-13T14:49:16.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:16 smithi103 ceph-mon[25851]: Deploying daemon osd.2 on smithi164 2024-02-13T14:49:17.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:17 smithi164 ceph-mon[27636]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:17.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:17 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:17.573 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:17 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:17.573 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:17 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:17 smithi103 ceph-mon[25851]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:17 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:17 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:17.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:17 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:17.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:17 smithi158 ceph-mon[27628]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:17.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:17 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:17.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:17 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:17.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:17 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:19.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:19 smithi164 ceph-mon[27636]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:19.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:19 smithi103 ceph-mon[25851]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:19.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:19 smithi158 ceph-mon[27628]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:21.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:21 smithi164 ceph-mon[27636]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:21.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:21 smithi103 ceph-mon[25851]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:21.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:21 smithi158 ceph-mon[27628]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:22.879 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:22 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:22.879 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:22 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:22.879 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:22 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:22.879 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:22 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:22.879 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:22 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:22.879 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:22 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:22.879 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:22 smithi164 ceph-mon[27636]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:22 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:22 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:22 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:22 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:22 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:22 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:22 smithi103 ceph-mon[25851]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:22.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:22 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:22.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:22 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:22.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:22 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:22.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:22 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:22.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:22 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:22 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:22 smithi158 ceph-mon[27628]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:24.715 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 2 on host 'smithi164' 2024-02-13T14:49:25.306 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:25 smithi164 ceph-mon[27636]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:25.306 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:25 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:25.306 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:25 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:25.306 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:25 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:25.307 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:25 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:49:25.307 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:25 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:25.307 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:25 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:25.307 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:25 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:25.307 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:25 smithi164 ceph-mon[27636]: from='osd.2 [v2:172.21.15.164:6800/1896630575,v1:172.21.15.164:6801/1896630575]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T14:49:25.307 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:25 smithi164 ceph-mon[27636]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T14:49:25.307 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:25 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:25.308 DEBUG:teuthology.orchestra.run.smithi164:osd.2> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@osd.2.service 2024-02-13T14:49:25.311 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-02-13T14:49:25.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph osd stat -f json 2024-02-13T14:49:25.566 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:49:25.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:25 smithi103 ceph-mon[25851]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:25.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:25 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:25.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:25 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:25.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:25 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:25.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:25 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:49:25.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:25 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:25.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:25 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:25.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:25 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:25.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:25 smithi103 ceph-mon[25851]: from='osd.2 [v2:172.21.15.164:6800/1896630575,v1:172.21.15.164:6801/1896630575]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T14:49:25.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:25 smithi103 ceph-mon[25851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T14:49:25.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:25 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:25.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:25 smithi158 ceph-mon[27628]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:25.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:25 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:25.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:25 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:25.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:25 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:25.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:25 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:49:25.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:25 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:25.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:25 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:25.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:25 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:25.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:25 smithi158 ceph-mon[27628]: from='osd.2 [v2:172.21.15.164:6800/1896630575,v1:172.21.15.164:6801/1896630575]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T14:49:25.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:25 smithi158 ceph-mon[27628]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-13T14:49:25.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:25 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:26.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:26 smithi164 ceph-mon[27636]: Adjusting osd_memory_target on smithi164 to 16064M 2024-02-13T14:49:26.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:26 smithi164 ceph-mon[27636]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T14:49:26.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:26 smithi164 ceph-mon[27636]: osdmap e16: 3 total, 2 up, 3 in 2024-02-13T14:49:26.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:26 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:26.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:26 smithi164 ceph-mon[27636]: from='osd.2 [v2:172.21.15.164:6800/1896630575,v1:172.21.15.164:6801/1896630575]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-02-13T14:49:26.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:26 smithi164 ceph-mon[27636]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-02-13T14:49:26.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:26 smithi103 ceph-mon[25851]: Adjusting osd_memory_target on smithi164 to 16064M 2024-02-13T14:49:26.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:26 smithi103 ceph-mon[25851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T14:49:26.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:26 smithi103 ceph-mon[25851]: osdmap e16: 3 total, 2 up, 3 in 2024-02-13T14:49:26.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:26 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:26.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:26 smithi103 ceph-mon[25851]: from='osd.2 [v2:172.21.15.164:6800/1896630575,v1:172.21.15.164:6801/1896630575]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-02-13T14:49:26.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:26 smithi103 ceph-mon[25851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-02-13T14:49:26.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:26 smithi158 ceph-mon[27628]: Adjusting osd_memory_target on smithi164 to 16064M 2024-02-13T14:49:26.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:26 smithi158 ceph-mon[27628]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-13T14:49:26.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:26 smithi158 ceph-mon[27628]: osdmap e16: 3 total, 2 up, 3 in 2024-02-13T14:49:26.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:26 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:26.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:26 smithi158 ceph-mon[27628]: from='osd.2 [v2:172.21.15.164:6800/1896630575,v1:172.21.15.164:6801/1896630575]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-02-13T14:49:26.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:26 smithi158 ceph-mon[27628]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-02-13T14:49:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:49:27.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:27 smithi164 ceph-mon[27636]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:27.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:27 smithi164 ceph-mon[27636]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-02-13T14:49:27.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:27 smithi164 ceph-mon[27636]: osdmap e17: 3 total, 2 up, 3 in 2024-02-13T14:49:27.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:27 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:27.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:27 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:27.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:27 smithi164 ceph-mon[27636]: from='osd.2 ' entity='osd.2' 2024-02-13T14:49:27.641 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:49:27 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2[34235]: 2024-02-13T14:49:27.211+0000 7f4596404700 -1 osd.2 0 waiting for initial osdmap 2024-02-13T14:49:27.641 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:49:27 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2[34235]: 2024-02-13T14:49:27.214+0000 7f458d9ee700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-13T14:49:27.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:27 smithi103 ceph-mon[25851]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:27.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:27 smithi103 ceph-mon[25851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-02-13T14:49:27.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:27 smithi103 ceph-mon[25851]: osdmap e17: 3 total, 2 up, 3 in 2024-02-13T14:49:27.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:27 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:27.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:27 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:27.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:27 smithi103 ceph-mon[25851]: from='osd.2 ' entity='osd.2' 2024-02-13T14:49:27.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:27 smithi158 ceph-mon[27628]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:27.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:27 smithi158 ceph-mon[27628]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-02-13T14:49:27.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:27 smithi158 ceph-mon[27628]: osdmap e17: 3 total, 2 up, 3 in 2024-02-13T14:49:27.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:27 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:27.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:27 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:27.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:27 smithi158 ceph-mon[27628]: from='osd.2 ' entity='osd.2' 2024-02-13T14:49:27.917 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":17,"num_osds":3,"num_up_osds":2,"osd_up_since":1707835740,"num_in_osds":3,"osd_in_since":1707835747,"num_remapped_pgs":0} 2024-02-13T14:49:28.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:28 smithi103 ceph-mon[25851]: purged_snaps scrub starts 2024-02-13T14:49:28.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:28 smithi103 ceph-mon[25851]: purged_snaps scrub ok 2024-02-13T14:49:28.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:28 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/3264127302' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T14:49:28.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:28 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:28.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:28 smithi103 ceph-mon[25851]: osd.2 [v2:172.21.15.164:6800/1896630575,v1:172.21.15.164:6801/1896630575] boot 2024-02-13T14:49:28.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:28 smithi103 ceph-mon[25851]: osdmap e18: 3 total, 3 up, 3 in 2024-02-13T14:49:28.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:28 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:28.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:28 smithi164 ceph-mon[27636]: purged_snaps scrub starts 2024-02-13T14:49:28.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:28 smithi164 ceph-mon[27636]: purged_snaps scrub ok 2024-02-13T14:49:28.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:28 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/3264127302' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T14:49:28.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:28 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:28.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:28 smithi164 ceph-mon[27636]: osd.2 [v2:172.21.15.164:6800/1896630575,v1:172.21.15.164:6801/1896630575] boot 2024-02-13T14:49:28.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:28 smithi164 ceph-mon[27636]: osdmap e18: 3 total, 3 up, 3 in 2024-02-13T14:49:28.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:28 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:28.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:28 smithi158 ceph-mon[27628]: purged_snaps scrub starts 2024-02-13T14:49:28.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:28 smithi158 ceph-mon[27628]: purged_snaps scrub ok 2024-02-13T14:49:28.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:28 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/3264127302' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T14:49:28.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:28 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:28.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:28 smithi158 ceph-mon[27628]: osd.2 [v2:172.21.15.164:6800/1896630575,v1:172.21.15.164:6801/1896630575] boot 2024-02-13T14:49:28.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:28 smithi158 ceph-mon[27628]: osdmap e18: 3 total, 3 up, 3 in 2024-02-13T14:49:28.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:28 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:49:28.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph osd stat -f json 2024-02-13T14:49:29.503 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:49:29.624 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:29 smithi103 ceph-mon[25851]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:29.625 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:29 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-13T14:49:29.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:29 smithi164 ceph-mon[27636]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:29.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:29 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-13T14:49:29.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:29 smithi158 ceph-mon[27628]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-13T14:49:29.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:29 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-13T14:49:30.491 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:30 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-13T14:49:30.491 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:30 smithi103 ceph-mon[25851]: osdmap e19: 3 total, 3 up, 3 in 2024-02-13T14:49:30.491 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:30 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T14:49:30.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:30 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-13T14:49:30.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:30 smithi164 ceph-mon[27636]: osdmap e19: 3 total, 3 up, 3 in 2024-02-13T14:49:30.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:30 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T14:49:30.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:30 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-13T14:49:30.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:30 smithi158 ceph-mon[27628]: osdmap e19: 3 total, 3 up, 3 in 2024-02-13T14:49:30.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:30 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-13T14:49:30.800 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:49:30 smithi103 sudo[40522]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme4n1 2024-02-13T14:49:30.800 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:49:30 smithi103 sudo[40522]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:30.800 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:49:30 smithi103 sudo[40522]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:30.800 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:49:30 smithi103 sudo[40522]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:31.155 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:49:30 smithi103 sudo[40553]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-02-13T14:49:31.156 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:49:30 smithi103 sudo[40553]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:31.156 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:49:30 smithi103 sudo[40553]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:31.157 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:49:30 smithi103 sudo[40553]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:31.436 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:31 smithi158 ceph-mon[27628]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:31.437 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:31 smithi158 ceph-mon[27628]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T14:49:31.437 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-13T14:49:31.437 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:31 smithi158 ceph-mon[27628]: osdmap e20: 3 total, 3 up, 3 in 2024-02-13T14:49:31.437 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.437 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.437 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.437 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:31 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.437 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:49:31 smithi158 sudo[34468]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme4n1 2024-02-13T14:49:31.437 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:49:31 smithi158 sudo[34468]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:31.437 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:49:31 smithi158 sudo[34468]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:31.437 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:49:31 smithi158 sudo[34468]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:31.438 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:49:31 smithi158 sudo[34471]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-02-13T14:49:31.522 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:31 smithi103 ceph-mon[25851]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:31.522 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:31 smithi103 ceph-mon[25851]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T14:49:31.522 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-13T14:49:31.523 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:31 smithi103 ceph-mon[25851]: osdmap e20: 3 total, 3 up, 3 in 2024-02-13T14:49:31.523 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.523 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.523 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.523 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:31 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:31 smithi164 ceph-mon[27636]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:31.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:31 smithi164 ceph-mon[27636]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-13T14:49:31.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-13T14:49:31.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:31 smithi164 ceph-mon[27636]: osdmap e20: 3 total, 3 up, 3 in 2024-02-13T14:49:31.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:31 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:31.830 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:49:31.928 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:49:31 smithi158 sudo[34471]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:31.928 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:49:31 smithi158 sudo[34471]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:31.928 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:49:31 smithi158 sudo[34471]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:31.972 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:49:31 smithi164 sudo[35302]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme4n1 2024-02-13T14:49:31.972 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:49:31 smithi164 sudo[35302]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:31.972 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:49:31 smithi164 sudo[35302]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:31.972 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:49:31 smithi164 sudo[35302]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:32.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 sudo[40625]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-13T14:49:32.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 sudo[40625]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:32.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 sudo[40625]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:32.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 sudo[40625]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:32.384 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":21,"num_osds":3,"num_up_osds":3,"osd_up_since":1707835768,"num_in_osds":3,"osd_in_since":1707835747,"num_remapped_pgs":0} 2024-02-13T14:49:32.385 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-13T14:49:32.390 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:49:31 smithi164 sudo[35359]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-02-13T14:49:32.390 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:49:31 smithi164 sudo[35359]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:32.390 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:49:31 smithi164 sudo[35359]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:32.390 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:49:31 smithi164 sudo[35359]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:32.629 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:49:32.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 sudo[40646]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-13T14:49:32.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 sudo[40646]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:32.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 sudo[40646]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:32.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 sudo[40646]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:32.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 ceph-mon[25851]: osdmap e21: 3 total, 3 up, 3 in 2024-02-13T14:49:32.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2421751757' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T14:49:32.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 ceph-mon[25851]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-02-13T14:49:32.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.656 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:32 smithi103 ceph-mon[25851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T14:49:32.773 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 ceph-mon[27636]: osdmap e21: 3 total, 3 up, 3 in 2024-02-13T14:49:32.773 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.773 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2421751757' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T14:49:32.773 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.773 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.773 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.773 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 ceph-mon[27636]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-02-13T14:49:32.774 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.774 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 ceph-mon[27636]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T14:49:32.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:32 smithi158 ceph-mon[27628]: osdmap e21: 3 total, 3 up, 3 in 2024-02-13T14:49:32.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:32 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:32 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2421751757' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-13T14:49:32.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:32 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:32 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:32 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:32 smithi158 ceph-mon[27628]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-02-13T14:49:32.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:32 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:32.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:32 smithi158 ceph-mon[27628]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T14:49:33.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 sudo[35362]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-13T14:49:33.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 sudo[35362]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:33.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 sudo[35362]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:33.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:32 smithi164 sudo[35362]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:33.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 sudo[35365]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-13T14:49:33.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 sudo[35365]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:33.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 sudo[35365]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:33.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 sudo[35365]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:33.542 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 sudo[34474]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-13T14:49:33.543 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 sudo[34474]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:33.543 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 sudo[34474]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:33.543 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 sudo[34474]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:33.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: Detected new or changed devices on smithi164 2024-02-13T14:49:33.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:49:33.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:33 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:49:33.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: Detected new or changed devices on smithi164 2024-02-13T14:49:33.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:33.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T14:49:33.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:49:33.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:49:33.865 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:49:33.866 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:33.866 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T14:49:33.866 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:49:33.866 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:49:33.866 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:49:33.866 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T14:49:33.866 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:33.866 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:49:33.866 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:49:33.866 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:33 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:49:33.893 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 sudo[34484]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 sudo[34484]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 sudo[34484]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 sudo[34484]: pam_unix(sudo:session): session closed for user root 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: Detected new or changed devices on smithi164 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T14:49:33.894 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:49:33.895 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:49:33.895 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:49:33.895 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T14:49:33.895 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:33.895 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:49:33.895 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:49:33.895 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:33 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:49:34.381 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:49:34.381 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":21,"fsid":"7acdb230-ca7e-11ee-95b9-87774f69a715","created":"2024-02-13T14:45:53.543989+0000","modified":"2024-02-13T14:49:31.303660+0000","last_up_change":"2024-02-13T14:49:28.214553+0000","last_in_change":"2024-02-13T14:49:07.192460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-13T14:49:28.544313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c2c5b1d3-1c27-44d0-87a6-45b988205060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6803","nonce":2605990222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6805","nonce":2605990222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6809","nonce":2605990222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6807","nonce":2605990222}]},"public_addr":"172.21.15.103:6803/2605990222","cluster_addr":"172.21.15.103:6805/2605990222","heartbeat_back_addr":"172.21.15.103:6809/2605990222","heartbeat_front_addr":"172.21.15.103:6807/2605990222","state":["exists","up"]},{"osd":1,"uuid":"5bd2fdeb-bc00-434a-9e38-831a4e8be19f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6801","nonce":2266577857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6803","nonce":2266577857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6807","nonce":2266577857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6805","nonce":2266577857}]},"public_addr":"172.21.15.158:6801/2266577857","cluster_addr":"172.21.15.158:6803/2266577857","heartbeat_back_addr":"172.21.15.158:6807/2266577857","heartbeat_front_addr":"172.21.15.158:6805/2266577857","state":["exists","up"]},{"osd":2,"uuid":"f338f379-8817-43f4-a81e-6a851fd7196b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6801","nonce":1896630575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6803","nonce":1896630575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6807","nonce":1896630575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6805","nonce":1896630575}]},"public_addr":"172.21.15.164:6801/1896630575","cluster_addr":"172.21.15.164:6803/1896630575","heartbeat_back_addr":"172.21.15.164:6807/1896630575","heartbeat_front_addr":"172.21.15.164:6805/1896630575","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T14:48:34.487663+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T14:48:58.795686+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T14:49:25.757401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/2622031320":"2024-02-14T14:46:56.207338+0000","172.21.15.103:0/1541801909":"2024-02-14T14:46:56.207338+0000","172.21.15.103:6801/105932927":"2024-02-14T14:46:29.826600+0000","172.21.15.103:6800/105932927":"2024-02-14T14:46:29.826600+0000","172.21.15.103:6801/1391396974":"2024-02-14T14:46:56.207338+0000","172.21.15.103:6800/1391396974":"2024-02-14T14:46:56.207338+0000","172.21.15.103:0/1139541410":"2024-02-14T14:46:29.826600+0000","172.21.15.103:0/1180240934":"2024-02-14T14:46:29.826600+0000","172.21.15.103:0/571209747":"2024-02-14T14:46:29.826600+0000","172.21.15.103:0/2554106681":"2024-02-14T14:46:56.207338+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-13T14:49:34.763 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:34 smithi158 ceph-mon[27628]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T14:49:34.763 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:34 smithi158 ceph-mon[27628]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-13T14:49:34.763 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:34 smithi158 ceph-mon[27628]: Cluster is now healthy 2024-02-13T14:49:34.763 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:34 smithi158 ceph-mon[27628]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T14:49:34.763 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:34 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:34.763 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:34 smithi158 ceph-mon[27628]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:34.763 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:34 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2866801059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T14:49:34.858 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:34 smithi164 ceph-mon[27636]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T14:49:34.858 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:34 smithi164 ceph-mon[27636]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-13T14:49:34.859 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:34 smithi164 ceph-mon[27636]: Cluster is now healthy 2024-02-13T14:49:34.859 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:34 smithi164 ceph-mon[27636]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T14:49:34.859 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:34 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:34.859 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:34 smithi164 ceph-mon[27636]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:34.859 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:34 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2866801059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T14:49:34.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:34 smithi103 ceph-mon[25851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-13T14:49:34.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:34 smithi103 ceph-mon[25851]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-13T14:49:34.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:34 smithi103 ceph-mon[25851]: Cluster is now healthy 2024-02-13T14:49:34.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:34 smithi103 ceph-mon[25851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-13T14:49:34.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:34 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:34.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:34 smithi103 ceph-mon[25851]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:34.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:34 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2866801059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T14:49:34.981 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-13T14:49:28.544313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-13T14:49:34.981 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-13T14:49:35.217 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:49:37.023 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-02-13T14:49:37.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:37 smithi164 ceph-mon[27636]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:37.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:37 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:37.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:37 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:37.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:37 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:37.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:37 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/3615256498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-13T14:49:37.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:37 smithi103 ceph-mon[25851]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:37.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:37 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:37 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:37 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:37.600 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:37 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/3615256498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-13T14:49:37.627 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi103 2024-02-13T14:49:37.627 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi103=foo.a' 2024-02-13T14:49:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:37 smithi158 ceph-mon[27628]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:37 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:37 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:37 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:37.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:37 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/3615256498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-13T14:49:37.877 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.c/config 2024-02-13T14:49:39.628 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled rgw.foo.a update... 2024-02-13T14:49:39.629 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:39 smithi164 ceph-mon[27636]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.8 KiB/s rd, 414 KiB/s wr, 26 op/s 2024-02-13T14:49:39.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:39 smithi103 ceph-mon[25851]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.8 KiB/s rd, 414 KiB/s wr, 26 op/s 2024-02-13T14:49:39.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:39 smithi158 ceph-mon[27628]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.8 KiB/s rd, 414 KiB/s wr, 26 op/s 2024-02-13T14:49:40.141 DEBUG:teuthology.orchestra.run.smithi103:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@rgw.foo.a.service 2024-02-13T14:49:40.144 INFO:tasks.cephadm:Adding prometheus.a on smithi158 2024-02-13T14:49:40.144 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch apply prometheus '1;smithi158=a' 2024-02-13T14:49:40.379 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.c/config 2024-02-13T14:49:40.834 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi103=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:40.835 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: Saving service rgw.foo.a spec with placement smithi103=foo.a;count:1 2024-02-13T14:49:40.835 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:40.835 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:40.835 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:40.835 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:40.835 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:40.836 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T14:49:40.836 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T14:49:40.836 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:40.836 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:40.836 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: Deploying daemon rgw.foo.a on smithi103 2024-02-13T14:49:40.836 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:40 smithi164 ceph-mon[27636]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 322 KiB/s wr, 20 op/s 2024-02-13T14:49:40.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi103=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:40.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: Saving service rgw.foo.a spec with placement smithi103=foo.a;count:1 2024-02-13T14:49:40.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:40.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:40.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:40.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:40.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:40.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T14:49:40.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T14:49:40.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:40.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:40.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: Deploying daemon rgw.foo.a on smithi103 2024-02-13T14:49:40.906 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:40 smithi103 ceph-mon[25851]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 322 KiB/s wr, 20 op/s 2024-02-13T14:49:40.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi103=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:40.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: Saving service rgw.foo.a spec with placement smithi103=foo.a;count:1 2024-02-13T14:49:40.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:40.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:40.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:40.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:40.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:40.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-13T14:49:40.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-13T14:49:40.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:40.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:40.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: Deploying daemon rgw.foo.a on smithi103 2024-02-13T14:49:40.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:40 smithi158 ceph-mon[27628]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 322 KiB/s wr, 20 op/s 2024-02-13T14:49:41.887 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:49:41 smithi103 systemd[1]: Starting Ceph rgw.foo.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:49:41.970 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled prometheus update... 2024-02-13T14:49:42.154 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:49:41 smithi103 podman[41350]: 2024-02-13 14:49:41.789489397 +0000 UTC m=+0.018670630 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f 2024-02-13T14:49:42.155 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:49:41 smithi103 podman[41350]: 2024-02-13 14:49:41.898299961 +0000 UTC m=+0.127481221 container create 38f145a99eeb1b5c98d186b18b3270ef67a33e510e7cd718919bd8ed9d0f543b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-rgw-foo-a, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-02-13T14:49:42.155 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:49:42 smithi103 podman[41350]: 2024-02-13 14:49:42.100533239 +0000 UTC m=+0.329714461 container init 38f145a99eeb1b5c98d186b18b3270ef67a33e510e7cd718919bd8ed9d0f543b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T14:49:42.155 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:49:42 smithi103 podman[41350]: 2024-02-13 14:49:42.105617598 +0000 UTC m=+0.334798824 container start 38f145a99eeb1b5c98d186b18b3270ef67a33e510e7cd718919bd8ed9d0f543b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-rgw-foo-a, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-13T14:49:42.585 DEBUG:teuthology.orchestra.run.smithi158:prometheus.a> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@prometheus.a.service 2024-02-13T14:49:42.588 INFO:tasks.cephadm:Adding node-exporter.a on smithi103 2024-02-13T14:49:42.588 INFO:tasks.cephadm:Adding node-exporter.b on smithi158 2024-02-13T14:49:42.588 INFO:tasks.cephadm:Adding node-exporter.c on smithi164 2024-02-13T14:49:42.588 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch apply node-exporter '3;smithi103=a;smithi158=b;smithi164=c' 2024-02-13T14:49:42.654 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:49:42 smithi103 bash[41350]: 38f145a99eeb1b5c98d186b18b3270ef67a33e510e7cd718919bd8ed9d0f543b 2024-02-13T14:49:42.654 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:49:42 smithi103 systemd[1]: Started Ceph rgw.foo.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:49:42.822 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.c/config 2024-02-13T14:49:43.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi158=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:43.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: Saving service prometheus spec with placement smithi158=a;count:1 2024-02-13T14:49:43.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.9 KiB/s rd, 263 KiB/s wr, 16 op/s 2024-02-13T14:49:43.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:43.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:43.198 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:42 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:43.250 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi158=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:43.250 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: Saving service prometheus spec with placement smithi158=a;count:1 2024-02-13T14:49:43.250 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.250 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.9 KiB/s rd, 263 KiB/s wr, 16 op/s 2024-02-13T14:49:43.250 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.250 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.251 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.251 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.251 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.251 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:43.251 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:43.251 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:42 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:43.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi158=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:43.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: Saving service prometheus spec with placement smithi158=a;count:1 2024-02-13T14:49:43.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.9 KiB/s rd, 263 KiB/s wr, 16 op/s 2024-02-13T14:49:43.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:43.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:49:43.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:49:43.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:42 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:49:44.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:43 smithi164 ceph-mon[27636]: Saving service rgw.foo.a spec with placement smithi103=foo.a;count:1 2024-02-13T14:49:44.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:43 smithi164 ceph-mon[27636]: Checking dashboard <-> RGW credentials 2024-02-13T14:49:44.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:43 smithi164 ceph-mon[27636]: osdmap e22: 3 total, 3 up, 3 in 2024-02-13T14:49:44.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:43 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2189063993' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T14:49:44.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:43 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/3241222883' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T14:49:44.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:43 smithi103 ceph-mon[25851]: Saving service rgw.foo.a spec with placement smithi103=foo.a;count:1 2024-02-13T14:49:44.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:43 smithi103 ceph-mon[25851]: Checking dashboard <-> RGW credentials 2024-02-13T14:49:44.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:43 smithi103 ceph-mon[25851]: osdmap e22: 3 total, 3 up, 3 in 2024-02-13T14:49:44.300 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:43 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2189063993' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T14:49:44.301 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:43 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/3241222883' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T14:49:44.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:43 smithi158 ceph-mon[27628]: Saving service rgw.foo.a spec with placement smithi103=foo.a;count:1 2024-02-13T14:49:44.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:43 smithi158 ceph-mon[27628]: Checking dashboard <-> RGW credentials 2024-02-13T14:49:44.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:43 smithi158 ceph-mon[27628]: osdmap e22: 3 total, 3 up, 3 in 2024-02-13T14:49:44.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:43 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2189063993' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T14:49:44.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:43 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/3241222883' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-13T14:49:44.679 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled node-exporter update... 2024-02-13T14:49:45.248 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:44 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2189063993' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T14:49:45.248 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:44 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/3241222883' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T14:49:45.248 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:44 smithi164 ceph-mon[27636]: osdmap e23: 3 total, 3 up, 3 in 2024-02-13T14:49:45.248 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:44 smithi164 ceph-mon[27636]: pgmap v97: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 359 KiB/s wr, 22 op/s 2024-02-13T14:49:45.248 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:44 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:45.276 DEBUG:teuthology.orchestra.run.smithi103:node-exporter.a> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.a.service 2024-02-13T14:49:45.279 DEBUG:teuthology.orchestra.run.smithi158:node-exporter.b> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.b.service 2024-02-13T14:49:45.281 DEBUG:teuthology.orchestra.run.smithi164:node-exporter.c> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.c.service 2024-02-13T14:49:45.284 INFO:tasks.cephadm:Adding alertmanager.a on smithi103 2024-02-13T14:49:45.284 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch apply alertmanager '1;smithi103=a' 2024-02-13T14:49:45.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:44 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2189063993' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T14:49:45.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:44 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/3241222883' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T14:49:45.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:44 smithi103 ceph-mon[25851]: osdmap e23: 3 total, 3 up, 3 in 2024-02-13T14:49:45.304 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:44 smithi103 ceph-mon[25851]: pgmap v97: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 359 KiB/s wr, 22 op/s 2024-02-13T14:49:45.305 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:44 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:45.308 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:44 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2189063993' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T14:49:45.308 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:44 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/3241222883' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-13T14:49:45.308 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:44 smithi158 ceph-mon[27628]: osdmap e23: 3 total, 3 up, 3 in 2024-02-13T14:49:45.308 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:44 smithi158 ceph-mon[27628]: pgmap v97: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 359 KiB/s wr, 22 op/s 2024-02-13T14:49:45.308 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:44 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:45.556 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.c/config 2024-02-13T14:49:46.261 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:45 smithi164 ceph-mon[27636]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi103=a;smithi158=b;smithi164=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:46.262 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:45 smithi164 ceph-mon[27636]: Saving service node-exporter spec with placement smithi103=a;smithi158=b;smithi164=c;count:3 2024-02-13T14:49:46.262 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:45 smithi164 ceph-mon[27636]: osdmap e24: 3 total, 3 up, 3 in 2024-02-13T14:49:46.262 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:45 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T14:49:46.262 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:45 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T14:49:46.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:45 smithi103 ceph-mon[25851]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi103=a;smithi158=b;smithi164=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:46.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:45 smithi103 ceph-mon[25851]: Saving service node-exporter spec with placement smithi103=a;smithi158=b;smithi164=c;count:3 2024-02-13T14:49:46.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:45 smithi103 ceph-mon[25851]: osdmap e24: 3 total, 3 up, 3 in 2024-02-13T14:49:46.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:45 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T14:49:46.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:45 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T14:49:46.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:45 smithi158 ceph-mon[27628]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi103=a;smithi158=b;smithi164=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:46.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:45 smithi158 ceph-mon[27628]: Saving service node-exporter spec with placement smithi103=a;smithi158=b;smithi164=c;count:3 2024-02-13T14:49:46.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:45 smithi158 ceph-mon[27628]: osdmap e24: 3 total, 3 up, 3 in 2024-02-13T14:49:46.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:45 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T14:49:46.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:45 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-13T14:49:47.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:47 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T14:49:47.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:47 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T14:49:47.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:47 smithi164 ceph-mon[27636]: osdmap e25: 3 total, 3 up, 3 in 2024-02-13T14:49:47.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:47 smithi164 ceph-mon[27636]: pgmap v100: 65 pgs: 14 creating+peering, 1 active+clean, 50 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:47.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:47 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:47 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:47 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:47 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:47 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.279 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled alertmanager update... 2024-02-13T14:49:47.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:47 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T14:49:47.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:47 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T14:49:47.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:47 smithi103 ceph-mon[25851]: osdmap e25: 3 total, 3 up, 3 in 2024-02-13T14:49:47.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:47 smithi103 ceph-mon[25851]: pgmap v100: 65 pgs: 14 creating+peering, 1 active+clean, 50 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:47.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:47 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:47 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:47 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:47 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:47 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:47 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T14:49:47.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:47 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-13T14:49:47.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:47 smithi158 ceph-mon[27628]: osdmap e25: 3 total, 3 up, 3 in 2024-02-13T14:49:47.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:47 smithi158 ceph-mon[27628]: pgmap v100: 65 pgs: 14 creating+peering, 1 active+clean, 50 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:49:47.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:47 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:47 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:47 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:47 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:47 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:47.911 DEBUG:teuthology.orchestra.run.smithi103:alertmanager.a> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@alertmanager.a.service 2024-02-13T14:49:47.914 INFO:tasks.cephadm:Adding grafana.a on smithi158 2024-02-13T14:49:47.914 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph orch apply grafana '1;smithi158=a' 2024-02-13T14:49:48.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:48 smithi164 ceph-mon[27636]: osdmap e26: 3 total, 3 up, 3 in 2024-02-13T14:49:48.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:48 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T14:49:48.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:48 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T14:49:48.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:48 smithi164 ceph-mon[27636]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi103=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:48.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:48 smithi164 ceph-mon[27636]: Saving service alertmanager spec with placement smithi103=a;count:1 2024-02-13T14:49:48.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:48 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:48.158 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.c/config 2024-02-13T14:49:48.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:48 smithi103 ceph-mon[25851]: osdmap e26: 3 total, 3 up, 3 in 2024-02-13T14:49:48.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:48 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T14:49:48.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:48 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T14:49:48.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:48 smithi103 ceph-mon[25851]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi103=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:48.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:48 smithi103 ceph-mon[25851]: Saving service alertmanager spec with placement smithi103=a;count:1 2024-02-13T14:49:48.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:48 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:48.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:48 smithi158 ceph-mon[27628]: osdmap e26: 3 total, 3 up, 3 in 2024-02-13T14:49:48.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:48 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T14:49:48.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:48 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-13T14:49:48.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:48 smithi158 ceph-mon[27628]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi103=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:48.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:48 smithi158 ceph-mon[27628]: Saving service alertmanager spec with placement smithi103=a;count:1 2024-02-13T14:49:48.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:48 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:49.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:49 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T14:49:49.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:49 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T14:49:49.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:49 smithi164 ceph-mon[27636]: osdmap e27: 3 total, 3 up, 3 in 2024-02-13T14:49:49.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:49 smithi164 ceph-mon[27636]: pgmap v103: 97 pgs: 42 active+clean, 25 creating+peering, 30 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-02-13T14:49:49.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:49 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T14:49:49.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:49 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T14:49:49.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:49 smithi103 ceph-mon[25851]: osdmap e27: 3 total, 3 up, 3 in 2024-02-13T14:49:49.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:49 smithi103 ceph-mon[25851]: pgmap v103: 97 pgs: 42 active+clean, 25 creating+peering, 30 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-02-13T14:49:49.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:49 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T14:49:49.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:49 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-13T14:49:49.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:49 smithi158 ceph-mon[27628]: osdmap e27: 3 total, 3 up, 3 in 2024-02-13T14:49:49.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:49 smithi158 ceph-mon[27628]: pgmap v103: 97 pgs: 42 active+clean, 25 creating+peering, 30 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-02-13T14:49:49.998 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled grafana update... 2024-02-13T14:49:50.283 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:50 smithi164 ceph-mon[27636]: osdmap e28: 3 total, 3 up, 3 in 2024-02-13T14:49:50.283 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:50 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T14:49:50.283 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:50 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T14:49:50.283 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:50 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:50.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:50 smithi103 ceph-mon[25851]: osdmap e28: 3 total, 3 up, 3 in 2024-02-13T14:49:50.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:50 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T14:49:50.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:50 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T14:49:50.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:50 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:50.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:50 smithi158 ceph-mon[27628]: osdmap e28: 3 total, 3 up, 3 in 2024-02-13T14:49:50.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:50 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T14:49:50.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:50 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-13T14:49:50.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:50 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:50.550 DEBUG:teuthology.orchestra.run.smithi158:grafana.a> sudo journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@grafana.a.service 2024-02-13T14:49:50.553 INFO:tasks.cephadm:Setting up client nodes... 2024-02-13T14:49:50.553 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-13T14:49:50.802 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:49:51.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:51 smithi103 ceph-mon[25851]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi158=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:51.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:51 smithi103 ceph-mon[25851]: Saving service grafana spec with placement smithi158=a;count:1 2024-02-13T14:49:51.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:51 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T14:49:51.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:51 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T14:49:51.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:51 smithi103 ceph-mon[25851]: osdmap e29: 3 total, 3 up, 3 in 2024-02-13T14:49:51.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:51 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T14:49:51.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:51 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T14:49:51.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:51 smithi103 ceph-mon[25851]: pgmap v106: 129 pgs: 77 active+clean, 11 creating+peering, 41 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-13T14:49:51.188 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:51 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T14:49:51.189 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:51 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T14:49:51.189 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:51 smithi103 ceph-mon[25851]: osdmap e30: 3 total, 3 up, 3 in 2024-02-13T14:49:51.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:51 smithi164 ceph-mon[27636]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi158=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:51.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:51 smithi164 ceph-mon[27636]: Saving service grafana spec with placement smithi158=a;count:1 2024-02-13T14:49:51.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:51 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T14:49:51.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:51 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T14:49:51.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:51 smithi164 ceph-mon[27636]: osdmap e29: 3 total, 3 up, 3 in 2024-02-13T14:49:51.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:51 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T14:49:51.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:51 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T14:49:51.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:51 smithi164 ceph-mon[27636]: pgmap v106: 129 pgs: 77 active+clean, 11 creating+peering, 41 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-13T14:49:51.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:51 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T14:49:51.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:51 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T14:49:51.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:51 smithi164 ceph-mon[27636]: osdmap e30: 3 total, 3 up, 3 in 2024-02-13T14:49:51.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:51 smithi158 ceph-mon[27628]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi158=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:49:51.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:51 smithi158 ceph-mon[27628]: Saving service grafana spec with placement smithi158=a;count:1 2024-02-13T14:49:51.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:51 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T14:49:51.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:51 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-13T14:49:51.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:51 smithi158 ceph-mon[27628]: osdmap e29: 3 total, 3 up, 3 in 2024-02-13T14:49:51.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:51 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T14:49:51.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:51 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-13T14:49:51.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:51 smithi158 ceph-mon[27628]: pgmap v106: 129 pgs: 77 active+clean, 11 creating+peering, 41 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-13T14:49:51.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:51 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/499611951' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T14:49:51.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:51 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2852379207' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-13T14:49:51.429 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:51 smithi158 ceph-mon[27628]: osdmap e30: 3 total, 3 up, 3 in 2024-02-13T14:49:52.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:52 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:52 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:52 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:52 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:52 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.361 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:52 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:52 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:52 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:52 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:52 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:52 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:52 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:52 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:52.756 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-02-13T14:49:52.756 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCQgctlmkTfLBAAKzgqBjyX1TRa8OVDmh4+Hw== 2024-02-13T14:49:53.313 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:53 smithi103 ceph-mon[25851]: Deploying daemon prometheus.a on smithi158 2024-02-13T14:49:53.313 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:53 smithi103 ceph-mon[25851]: pgmap v108: 129 pgs: 99 active+clean, 11 creating+peering, 19 unknown; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 34 KiB/s rd, 3.3 KiB/s wr, 82 op/s 2024-02-13T14:49:53.313 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:53 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/543746987' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:49:53.313 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:53 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/543746987' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T14:49:53.314 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:49:53.315 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-13T14:49:53.315 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-13T14:49:53.358 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-13T14:49:53.383 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:53 smithi158 ceph-mon[27628]: Deploying daemon prometheus.a on smithi158 2024-02-13T14:49:53.383 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:53 smithi158 ceph-mon[27628]: pgmap v108: 129 pgs: 99 active+clean, 11 creating+peering, 19 unknown; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 34 KiB/s rd, 3.3 KiB/s wr, 82 op/s 2024-02-13T14:49:53.383 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:53 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/543746987' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:49:53.383 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:53 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/543746987' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T14:49:53.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:53 smithi164 ceph-mon[27636]: Deploying daemon prometheus.a on smithi158 2024-02-13T14:49:53.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:53 smithi164 ceph-mon[27636]: pgmap v108: 129 pgs: 99 active+clean, 11 creating+peering, 19 unknown; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 34 KiB/s rd, 3.3 KiB/s wr, 82 op/s 2024-02-13T14:49:53.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:53 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/543746987' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:49:53.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:53 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/543746987' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T14:49:54.757 INFO:teuthology.orchestra.run.smithi158.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.b/config 2024-02-13T14:49:55.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:55 smithi164 ceph-mon[27636]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 74 KiB/s rd, 6.5 KiB/s wr, 178 op/s 2024-02-13T14:49:55.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:55 smithi103 ceph-mon[25851]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 74 KiB/s rd, 6.5 KiB/s wr, 178 op/s 2024-02-13T14:49:55.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:55 smithi158 ceph-mon[27628]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 74 KiB/s rd, 6.5 KiB/s wr, 178 op/s 2024-02-13T14:49:57.383 INFO:teuthology.orchestra.run.smithi158.stdout:[client.1] 2024-02-13T14:49:57.383 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQCVgctlGgBUFhAA87u/5oPX4sW49vLE3bhfmg== 2024-02-13T14:49:57.383 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:57 smithi158 ceph-mon[27628]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 5.0 KiB/s wr, 146 op/s 2024-02-13T14:49:57.383 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:57 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:57.383 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:57 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:57.383 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:57 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:57.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:57 smithi164 ceph-mon[27636]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 5.0 KiB/s wr, 146 op/s 2024-02-13T14:49:57.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:57.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:57.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:57 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:57.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:57 smithi103 ceph-mon[25851]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 5.0 KiB/s wr, 146 op/s 2024-02-13T14:49:57.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:57.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:57.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:57 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:49:58.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:58 smithi103 ceph-mon[25851]: from='client.? 172.21.15.158:0/1564335109' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:49:58.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:58 smithi103 ceph-mon[25851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:49:58.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:58 smithi103 ceph-mon[25851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T14:49:58.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:58 smithi164 ceph-mon[27636]: from='client.? 172.21.15.158:0/1564335109' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:49:58.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:58 smithi164 ceph-mon[27636]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:49:58.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:58 smithi164 ceph-mon[27636]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T14:49:58.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:58 smithi158 ceph-mon[27628]: from='client.? 172.21.15.158:0/1564335109' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:49:58.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:58 smithi158 ceph-mon[27628]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:49:58.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:58 smithi158 ceph-mon[27628]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T14:49:59.091 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-13T14:49:59.092 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-13T14:49:59.092 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-13T14:49:59.174 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-13T14:49:59.429 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.c/config 2024-02-13T14:49:59.536 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:49:59 smithi164 ceph-mon[27636]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 57 KiB/s rd, 4.4 KiB/s wr, 137 op/s 2024-02-13T14:49:59.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:49:59 smithi103 ceph-mon[25851]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 57 KiB/s rd, 4.4 KiB/s wr, 137 op/s 2024-02-13T14:49:59.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:49:59 smithi158 ceph-mon[27628]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 57 KiB/s rd, 4.4 KiB/s wr, 137 op/s 2024-02-13T14:50:00.542 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:00 smithi164 ceph-mon[27636]: overall HEALTH_OK 2024-02-13T14:50:00.563 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:00 smithi158 ceph-mon[27628]: overall HEALTH_OK 2024-02-13T14:50:00.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:00 smithi103 ceph-mon[25851]: overall HEALTH_OK 2024-02-13T14:50:01.319 INFO:teuthology.orchestra.run.smithi164.stdout:[client.2] 2024-02-13T14:50:01.319 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCZgctlFXphEhAATudJD6oKVxHcSZ/b9rIvXg== 2024-02-13T14:50:01.493 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:01 smithi158 ceph-mon[27628]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.6 KiB/s wr, 112 op/s 2024-02-13T14:50:01.589 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:01 smithi164 ceph-mon[27636]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.6 KiB/s wr, 112 op/s 2024-02-13T14:50:01.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:01 smithi103 ceph-mon[25851]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.6 KiB/s wr, 112 op/s 2024-02-13T14:50:01.883 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-02-13T14:50:01.884 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-02-13T14:50:01.884 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-02-13T14:50:01.927 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-13T14:50:01.927 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-13T14:50:01.927 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph mgr dump --format=json 2024-02-13T14:50:02.179 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:02.633 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:02 smithi103 ceph-mon[25851]: from='client.? 172.21.15.164:0/566828376' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:50:02.634 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:02 smithi103 ceph-mon[25851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:50:02.634 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:02 smithi103 ceph-mon[25851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T14:50:02.634 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:02 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:02.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:02 smithi164 ceph-mon[27636]: from='client.? 172.21.15.164:0/566828376' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:50:02.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:02 smithi164 ceph-mon[27636]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:50:02.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:02 smithi164 ceph-mon[27636]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T14:50:02.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:02 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:02.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:02 smithi158 ceph-mon[27628]: from='client.? 172.21.15.164:0/566828376' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:50:02.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:02 smithi158 ceph-mon[27628]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-13T14:50:02.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:02 smithi158 ceph-mon[27628]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-13T14:50:02.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:02 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:03.534 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:03 smithi103 ceph-mon[25851]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 3.2 KiB/s wr, 100 op/s 2024-02-13T14:50:03.580 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:03 smithi158 ceph-mon[27628]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 3.2 KiB/s wr, 100 op/s 2024-02-13T14:50:03.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:03 smithi164 ceph-mon[27636]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 3.2 KiB/s wr, 100 op/s 2024-02-13T14:50:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:50:04.136 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 podman[35667]: 2024-02-13 14:50:04.135136367 +0000 UTC m=+0.151580587 container create 6ce093964089f942d242ae8c15859302bab98e0d7fd6d9e8f71496f17cdc34a1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T14:50:04.394 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/945685280' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 podman[35667]: 2024-02-13 14:50:04.337813485 +0000 UTC m=+0.354257707 container init 6ce093964089f942d242ae8c15859302bab98e0d7fd6d9e8f71496f17cdc34a1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 podman[35667]: 2024-02-13 14:50:04.343267168 +0000 UTC m=+0.359711387 container start 6ce093964089f942d242ae8c15859302bab98e0d7fd6d9e8f71496f17cdc34a1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.367Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.367Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.367Z caller=main.go:570 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi158 (none))" 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.367Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.367Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.368Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.369Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.370Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.370Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.372Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-13T14:50:04.395 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.372Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.583µs 2024-02-13T14:50:04.396 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.372Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-13T14:50:04.396 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.373Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-13T14:50:04.396 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.373Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.577µs wal_replay_duration=280.268µs wbl_replay_duration=89ns total_replay_duration=316.97µs 2024-02-13T14:50:04.396 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.373Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-13T14:50:04.396 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.373Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-13T14:50:04.396 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.373Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-13T14:50:04.396 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.392Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.938343ms db_storage=714ns remote_storage=1.229µs web_handler=299ns query_engine=562ns scrape=386.254µs scrape_sd=63.738µs notify=720ns notify_sd=1.316µs rules=18.310833ms tracing=5.175µs 2024-02-13T14:50:04.396 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.392Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-13T14:50:04.396 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:50:04.392Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-13T14:50:04.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:04 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/945685280' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-13T14:50:04.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:04 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/945685280' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-13T14:50:04.663 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 bash[35667]: 6ce093964089f942d242ae8c15859302bab98e0d7fd6d9e8f71496f17cdc34a1 2024-02-13T14:50:04.664 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:50:04 smithi158 systemd[1]: Started Ceph prometheus.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:50:05.419 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:05 smithi103 ceph-mon[25851]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 66 op/s 2024-02-13T14:50:05.419 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:05 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:05.419 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:05 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:05.419 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:05 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:05.419 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:05 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-13T14:50:05.443 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":16,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1065156701},{"type":"v1","addr":"172.21.15.103:6801","nonce":1065156701}]},"active_addr":"172.21.15.103:6801/1065156701","active_change":"2024-02-13T14:46:56.207486+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.103:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":1383251712}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":132686394}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":1101394319}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":2700983865}]}]} 2024-02-13T14:50:05.449 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-13T14:50:05.449 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-13T14:50:05.449 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-13T14:50:05.571 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:05 smithi158 ceph-mon[27628]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 66 op/s 2024-02-13T14:50:05.571 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:05 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:05.572 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:05 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:05.572 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:05 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:05.572 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:05 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-13T14:50:05.572 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:05 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: ignoring --setuser ceph since I am not root 2024-02-13T14:50:05.572 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:05 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: ignoring --setgroup ceph since I am not root 2024-02-13T14:50:05.572 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:05 smithi158 ceph-mgr[29347]: -- 172.21.15.158:0/1347920009 <== mon.2 v2:172.21.15.158:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562e3acfa1a0 con 0x562e3acd0800 2024-02-13T14:50:05.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:05 smithi164 ceph-mon[27636]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 66 op/s 2024-02-13T14:50:05.573 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:05 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:05.573 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:05 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:05.573 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:05 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' 2024-02-13T14:50:05.573 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:05 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-13T14:50:05.687 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:05 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ignoring --setuser ceph since I am not root 2024-02-13T14:50:05.687 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:05 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ignoring --setgroup ceph since I am not root 2024-02-13T14:50:05.890 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:05 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: ignoring --setuser ceph since I am not root 2024-02-13T14:50:05.890 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:05 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: ignoring --setgroup ceph since I am not root 2024-02-13T14:50:05.890 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:05 smithi164 ceph-mgr[30128]: -- 172.21.15.164:0/2088380581 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ac8f6c21a0 con 0x55ac8f698c00 2024-02-13T14:50:05.900 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:06.122 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:06 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:06.088+0000 7fdc43be5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T14:50:06.324 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:06 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:06.133+0000 7f56967eb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T14:50:06.324 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:06 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:06.237+0000 7f56967eb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T14:50:06.390 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:06.085+0000 7f12dce3d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-13T14:50:06.390 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:06.179+0000 7f12dce3d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T14:50:06.428 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:06 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:06.184+0000 7fdc43be5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-13T14:50:06.624 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:06 smithi103 ceph-mon[25851]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-13T14:50:06.624 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:06 smithi103 ceph-mon[25851]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-02-13T14:50:06.642 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:06 smithi164 ceph-mon[27636]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-13T14:50:06.642 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:06 smithi164 ceph-mon[27636]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-02-13T14:50:06.821 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:06 smithi158 ceph-mon[27628]: from='mgr.14150 172.21.15.103:0/328108307' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-13T14:50:06.821 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:06 smithi158 ceph-mon[27628]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-02-13T14:50:06.936 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:06.838+0000 7f12dce3d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T14:50:06.936 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:06.934+0000 7f12dce3d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T14:50:07.151 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:06 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:06.818+0000 7fdc43be5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T14:50:07.151 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:06 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:06.911+0000 7fdc43be5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T14:50:07.151 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:07 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:07.053+0000 7fdc43be5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T14:50:07.151 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:07 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:07.148+0000 7fdc43be5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T14:50:07.236 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:06 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:06.905+0000 7f56967eb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-13T14:50:07.236 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:07 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:06.998+0000 7f56967eb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-13T14:50:07.236 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:07 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:07.139+0000 7f56967eb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T14:50:07.263 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:07 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:07.080+0000 7f12dce3d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-13T14:50:07.263 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:07 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:07.176+0000 7f12dce3d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T14:50:07.263 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:07 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:07.261+0000 7f12dce3d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T14:50:07.429 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:07 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:07.232+0000 7fdc43be5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T14:50:07.538 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:07 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:07.235+0000 7f56967eb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-13T14:50:07.539 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:07 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:07.327+0000 7f56967eb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-13T14:50:07.787 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:07 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:07.495+0000 7fdc43be5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T14:50:07.788 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:07 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:07.604+0000 7fdc43be5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T14:50:07.788 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:07 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:07.689+0000 7fdc43be5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T14:50:07.788 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:07 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:07.785+0000 7fdc43be5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T14:50:07.794 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:07 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:07.598+0000 7f56967eb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T14:50:07.794 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:07 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:07.709+0000 7f56967eb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T14:50:07.830 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:07 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:07.533+0000 7f12dce3d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-13T14:50:07.830 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:07 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:07.646+0000 7f12dce3d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-13T14:50:07.830 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:07 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:07.733+0000 7f12dce3d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T14:50:07.830 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:07 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:07.827+0000 7f12dce3d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T14:50:08.084 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:07 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:07.801+0000 7f56967eb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-13T14:50:08.084 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:07 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:07.896+0000 7f56967eb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-13T14:50:08.140 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:08 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:08.004+0000 7f12dce3d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T14:50:08.178 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:07 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:07.960+0000 7fdc43be5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T14:50:08.223 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:50:08.223 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":30,"fsid":"7acdb230-ca7e-11ee-95b9-87774f69a715","created":"2024-02-13T14:45:53.543989+0000","modified":"2024-02-13T14:49:51.018899+0000","last_up_change":"2024-02-13T14:49:28.214553+0000","last_in_change":"2024-02-13T14:49:07.192460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-13T14:49:28.544313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-13T14:49:42.377046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-02-13T14:49:44.287754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-02-13T14:49:46.248282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-02-13T14:49:48.300569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c2c5b1d3-1c27-44d0-87a6-45b988205060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6803","nonce":2605990222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6805","nonce":2605990222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6809","nonce":2605990222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6807","nonce":2605990222}]},"public_addr":"172.21.15.103:6803/2605990222","cluster_addr":"172.21.15.103:6805/2605990222","heartbeat_back_addr":"172.21.15.103:6809/2605990222","heartbeat_front_addr":"172.21.15.103:6807/2605990222","state":["exists","up"]},{"osd":1,"uuid":"5bd2fdeb-bc00-434a-9e38-831a4e8be19f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6801","nonce":2266577857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6803","nonce":2266577857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6807","nonce":2266577857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6805","nonce":2266577857}]},"public_addr":"172.21.15.158:6801/2266577857","cluster_addr":"172.21.15.158:6803/2266577857","heartbeat_back_addr":"172.21.15.158:6807/2266577857","heartbeat_front_addr":"172.21.15.158:6805/2266577857","state":["exists","up"]},{"osd":2,"uuid":"f338f379-8817-43f4-a81e-6a851fd7196b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6801","nonce":1896630575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6803","nonce":1896630575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6807","nonce":1896630575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6805","nonce":1896630575}]},"public_addr":"172.21.15.164:6801/1896630575","cluster_addr":"172.21.15.164:6803/1896630575","heartbeat_back_addr":"172.21.15.164:6807/1896630575","heartbeat_front_addr":"172.21.15.164:6805/1896630575","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T14:48:34.487663+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T14:48:58.795686+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T14:49:25.757401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/2622031320":"2024-02-14T14:46:56.207338+0000","172.21.15.103:0/1541801909":"2024-02-14T14:46:56.207338+0000","172.21.15.103:6801/105932927":"2024-02-14T14:46:29.826600+0000","172.21.15.103:6800/105932927":"2024-02-14T14:46:29.826600+0000","172.21.15.103:6801/1391396974":"2024-02-14T14:46:56.207338+0000","172.21.15.103:6800/1391396974":"2024-02-14T14:46:56.207338+0000","172.21.15.103:0/1139541410":"2024-02-14T14:46:29.826600+0000","172.21.15.103:0/1180240934":"2024-02-14T14:46:29.826600+0000","172.21.15.103:0/571209747":"2024-02-14T14:46:29.826600+0000","172.21.15.103:0/2554106681":"2024-02-14T14:46:56.207338+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-13T14:50:08.404 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:08 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/253213108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T14:50:08.405 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:08 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:08.081+0000 7f56967eb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-13T14:50:08.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:08 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/253213108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T14:50:08.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:08 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/253213108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T14:50:08.867 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-13T14:50:08.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-13T14:50:09.114 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:09.364 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:09 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:09.092+0000 7f56967eb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T14:50:09.364 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:09 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:09.233+0000 7f56967eb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T14:50:09.390 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:08 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:08.971+0000 7f12dce3d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T14:50:09.390 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:09.100+0000 7f12dce3d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T14:50:09.394 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:08 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:08.926+0000 7fdc43be5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-13T14:50:09.394 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:09 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:09.056+0000 7fdc43be5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-13T14:50:09.394 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:09 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:09.391+0000 7fdc43be5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T14:50:09.645 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:09 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:09.486+0000 7fdc43be5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T14:50:09.654 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:09 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:09.586+0000 7f56967eb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T14:50:09.721 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:09.434+0000 7f12dce3d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-13T14:50:09.721 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:09.524+0000 7f12dce3d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T14:50:09.721 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:09.718+0000 7f12dce3d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:50:09.928 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:09 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:09.691+0000 7fdc43be5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:50:09.928 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:09 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:09.888+0000 7fdc43be5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T14:50:09.967 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:09 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:09.679+0000 7f56967eb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-13T14:50:09.968 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:09 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:09.880+0000 7f56967eb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:50:10.140 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:09 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:09.902+0000 7f12dce3d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T14:50:10.243 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:10 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:10.070+0000 7f56967eb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-13T14:50:10.451 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:10 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:10.187+0000 7f12dce3d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:50:10.451 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:10 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:10.448+0000 7f12dce3d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T14:50:10.458 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:10 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:10.187+0000 7fdc43be5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:50:10.522 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:10 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:10.365+0000 7f56967eb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-13T14:50:10.881 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:10 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:10.630+0000 7f56967eb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T14:50:10.890 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:10 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:10.683+0000 7f12dce3d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T14:50:10.928 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:10 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:10.455+0000 7fdc43be5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-13T14:50:10.928 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:10 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:10.695+0000 7fdc43be5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T14:50:10.985 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:50:10.986 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":30,"fsid":"7acdb230-ca7e-11ee-95b9-87774f69a715","created":"2024-02-13T14:45:53.543989+0000","modified":"2024-02-13T14:49:51.018899+0000","last_up_change":"2024-02-13T14:49:28.214553+0000","last_in_change":"2024-02-13T14:49:07.192460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-13T14:49:28.544313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-13T14:49:42.377046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-02-13T14:49:44.287754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-02-13T14:49:46.248282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-02-13T14:49:48.300569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c2c5b1d3-1c27-44d0-87a6-45b988205060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6803","nonce":2605990222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6805","nonce":2605990222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6809","nonce":2605990222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2605990222},{"type":"v1","addr":"172.21.15.103:6807","nonce":2605990222}]},"public_addr":"172.21.15.103:6803/2605990222","cluster_addr":"172.21.15.103:6805/2605990222","heartbeat_back_addr":"172.21.15.103:6809/2605990222","heartbeat_front_addr":"172.21.15.103:6807/2605990222","state":["exists","up"]},{"osd":1,"uuid":"5bd2fdeb-bc00-434a-9e38-831a4e8be19f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6801","nonce":2266577857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6803","nonce":2266577857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6807","nonce":2266577857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":2266577857},{"type":"v1","addr":"172.21.15.158:6805","nonce":2266577857}]},"public_addr":"172.21.15.158:6801/2266577857","cluster_addr":"172.21.15.158:6803/2266577857","heartbeat_back_addr":"172.21.15.158:6807/2266577857","heartbeat_front_addr":"172.21.15.158:6805/2266577857","state":["exists","up"]},{"osd":2,"uuid":"f338f379-8817-43f4-a81e-6a851fd7196b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6801","nonce":1896630575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6803","nonce":1896630575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6807","nonce":1896630575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":1896630575},{"type":"v1","addr":"172.21.15.164:6805","nonce":1896630575}]},"public_addr":"172.21.15.164:6801/1896630575","cluster_addr":"172.21.15.164:6803/1896630575","heartbeat_back_addr":"172.21.15.164:6807/1896630575","heartbeat_front_addr":"172.21.15.164:6805/1896630575","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T14:48:34.487663+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T14:48:58.795686+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-13T14:49:25.757401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/2622031320":"2024-02-14T14:46:56.207338+0000","172.21.15.103:0/1541801909":"2024-02-14T14:46:56.207338+0000","172.21.15.103:6801/105932927":"2024-02-14T14:46:29.826600+0000","172.21.15.103:6800/105932927":"2024-02-14T14:46:29.826600+0000","172.21.15.103:6801/1391396974":"2024-02-14T14:46:56.207338+0000","172.21.15.103:6800/1391396974":"2024-02-14T14:46:56.207338+0000","172.21.15.103:0/1139541410":"2024-02-14T14:46:29.826600+0000","172.21.15.103:0/1180240934":"2024-02-14T14:46:29.826600+0000","172.21.15.103:0/571209747":"2024-02-14T14:46:29.826600+0000","172.21.15.103:0/2554106681":"2024-02-14T14:46:56.207338+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-13T14:50:11.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:11 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/182790965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T14:50:11.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:10 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:10.878+0000 7f56967eb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-13T14:50:11.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:11 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/182790965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T14:50:11.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:11 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/182790965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-13T14:50:11.528 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-13T14:50:11.529 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-13T14:50:11.529 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-13T14:50:11.783 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:11.795 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:11.802 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:11.890 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:11 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:11.481+0000 7f12dce3d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T14:50:11.904 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:11 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:11.728+0000 7f56967eb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T14:50:11.928 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:11 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:11.489+0000 7fdc43be5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-13T14:50:12.368 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:12.055+0000 7fdc43be5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T14:50:12.368 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:12.152+0000 7fdc43be5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T14:50:12.368 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:12.365+0000 7fdc43be5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T14:50:12.390 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:12.080+0000 7f12dce3d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T14:50:12.390 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:12.180+0000 7f12dce3d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T14:50:12.443 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:12.350+0000 7f56967eb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-13T14:50:12.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: Standby manager daemon x restarted 2024-02-13T14:50:12.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: Standby manager daemon x started 2024-02-13T14:50:12.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: Standby manager daemon z restarted 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: Standby manager daemon z started 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-mon[27628]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: [13/Feb/2024:14:50:12] ENGINE Bus STARTING 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: CherryPy Checker: 2024-02-13T14:50:12.679 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: The Application mounted at '' has an empty config. 2024-02-13T14:50:12.680 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: 2024-02-13T14:50:12.680 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: [13/Feb/2024:14:50:12] ENGINE Serving on http://:::9283 2024-02-13T14:50:12.680 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:50:12 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x[29324]: [13/Feb/2024:14:50:12] ENGINE Bus STARTED 2024-02-13T14:50:12.795 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:12.459+0000 7f56967eb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-13T14:50:12.795 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:12.686+0000 7f56967eb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T14:50:12.795 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: Standby manager daemon x restarted 2024-02-13T14:50:12.795 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: Standby manager daemon x started 2024-02-13T14:50:12.795 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T14:50:12.795 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:50:12.795 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T14:50:12.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:50:12.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: Standby manager daemon z restarted 2024-02-13T14:50:12.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: Standby manager daemon z started 2024-02-13T14:50:12.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-13T14:50:12.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:50:12.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-13T14:50:12.796 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-mon[25851]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:50:12.822 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: Standby manager daemon x restarted 2024-02-13T14:50:12.822 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: Standby manager daemon x started 2024-02-13T14:50:12.822 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-13T14:50:12.822 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:50:12.822 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-13T14:50:12.823 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.158:0/247180911' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:50:12.823 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: Standby manager daemon z restarted 2024-02-13T14:50:12.823 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: Standby manager daemon z started 2024-02-13T14:50:12.823 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-13T14:50:12.823 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-13T14:50:12.823 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-13T14:50:12.823 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-mon[27636]: from='mgr.? 172.21.15.164:0/3453126802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-13T14:50:12.823 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:12.394+0000 7f12dce3d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-13T14:50:12.823 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: [13/Feb/2024:14:50:12] ENGINE Bus STARTING 2024-02-13T14:50:12.823 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: CherryPy Checker: 2024-02-13T14:50:12.823 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: The Application mounted at '' has an empty config. 2024-02-13T14:50:12.824 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: 2024-02-13T14:50:12.824 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: [13/Feb/2024:14:50:12] ENGINE Serving on http://:::9283 2024-02-13T14:50:12.824 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:50:12 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z[30068]: [13/Feb/2024:14:50:12] ENGINE Bus STARTED 2024-02-13T14:50:13.109 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:50:12] ENGINE Bus STARTING 2024-02-13T14:50:13.109 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: CherryPy Checker: 2024-02-13T14:50:13.109 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: The Application mounted at '' has an empty config. 2024-02-13T14:50:13.109 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:50:13.109 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:50:12] ENGINE Serving on http://:::9283 2024-02-13T14:50:13.109 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:50:12] ENGINE Bus STARTED 2024-02-13T14:50:13.717 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: Active manager daemon y restarted 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: Activating manager daemon y 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: osdmap e31: 3 total, 3 up, 3 in 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: mgrmap e19: y(active, starting, since 0.0102875s), standbys: x, z 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:50:13.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: Manager daemon y is now available 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]': finished 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]': finished 2024-02-13T14:50:13.719 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]': finished 2024-02-13T14:50:13.720 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:13.720 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:50:13.720 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:50:13.720 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:13.720 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T14:50:13.720 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:13 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T14:50:13.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-02-13T14:50:13.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: Active manager daemon y restarted 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: Activating manager daemon y 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: osdmap e31: 3 total, 3 up, 3 in 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: mgrmap e19: y(active, starting, since 0.0102875s), standbys: x, z 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T14:50:13.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: Manager daemon y is now available 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]': finished 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]': finished 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]': finished 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:13.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:50:13.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:50:13.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:13.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T14:50:13.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:13 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: Active manager daemon y restarted 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: Activating manager daemon y 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: osdmap e31: 3 total, 3 up, 3 in 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: mgrmap e19: y(active, starting, since 0.0102875s), standbys: x, z 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-13T14:50:13.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: Manager daemon y is now available 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]': finished 2024-02-13T14:50:13.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]: dispatch 2024-02-13T14:50:13.931 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]': finished 2024-02-13T14:50:13.931 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi158", "name": "osd_memory_target"}]': finished 2024-02-13T14:50:13.931 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:13.931 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:50:13.931 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:50:13.931 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:13.931 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-13T14:50:13.931 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:13 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-13T14:50:14.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: Updating smithi103:/etc/ceph/ceph.conf 2024-02-13T14:50:14.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: Updating smithi158:/etc/ceph/ceph.conf 2024-02-13T14:50:14.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: Updating smithi164:/etc/ceph/ceph.conf 2024-02-13T14:50:14.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: Updating smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:50:14.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:50:14.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: Updating smithi158:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: mgrmap e20: y(active, since 1.01782s), standbys: x, z 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:14 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi103", "caps": []}]: dispatch 2024-02-13T14:50:14.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: Updating smithi103:/etc/ceph/ceph.conf 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: Updating smithi158:/etc/ceph/ceph.conf 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: Updating smithi164:/etc/ceph/ceph.conf 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: Updating smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: Updating smithi158:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: mgrmap e20: y(active, since 1.01782s), standbys: x, z 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.642 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.642 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:14 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi103", "caps": []}]: dispatch 2024-02-13T14:50:14.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: Updating smithi103:/etc/ceph/ceph.conf 2024-02-13T14:50:14.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: Updating smithi158:/etc/ceph/ceph.conf 2024-02-13T14:50:14.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: Updating smithi164:/etc/ceph/ceph.conf 2024-02-13T14:50:14.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: Updating smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:50:14.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: Updating smithi158:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: mgrmap e20: y(active, since 1.01782s), standbys: x, z 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:14.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:14 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi103", "caps": []}]: dispatch 2024-02-13T14:50:15.348 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ::ffff:172.21.15.158 - - [13/Feb/2024:14:50:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-13T14:50:15.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: [13/Feb/2024:14:50:13] ENGINE Bus STARTING 2024-02-13T14:50:15.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:50:15.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: [13/Feb/2024:14:50:13] ENGINE Serving on http://172.21.15.103:8765 2024-02-13T14:50:15.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:15.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: Updating smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:50:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:50:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: [13/Feb/2024:14:50:13] ENGINE Serving on https://172.21.15.103:7151 2024-02-13T14:50:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: [13/Feb/2024:14:50:13] ENGINE Bus STARTED 2024-02-13T14:50:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:50:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:50:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi158", "caps": []}]: dispatch 2024-02-13T14:50:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:15.641 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:15 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:15.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: [13/Feb/2024:14:50:13] ENGINE Bus STARTING 2024-02-13T14:50:15.654 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: [13/Feb/2024:14:50:13] ENGINE Serving on http://172.21.15.103:8765 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: Updating smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: [13/Feb/2024:14:50:13] ENGINE Serving on https://172.21.15.103:7151 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: [13/Feb/2024:14:50:13] ENGINE Bus STARTED 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi158", "caps": []}]: dispatch 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:15.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:15 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: [13/Feb/2024:14:50:13] ENGINE Bus STARTING 2024-02-13T14:50:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-02-13T14:50:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: [13/Feb/2024:14:50:13] ENGINE Serving on http://172.21.15.103:8765 2024-02-13T14:50:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: Updating smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:50:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: Updating smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:50:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: [13/Feb/2024:14:50:13] ENGINE Serving on https://172.21.15.103:7151 2024-02-13T14:50:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: [13/Feb/2024:14:50:13] ENGINE Bus STARTED 2024-02-13T14:50:15.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: Updating smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-13T14:50:15.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:50:15.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi158", "caps": []}]: dispatch 2024-02-13T14:50:15.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:15.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:15.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:15 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.626 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:16 smithi103 ceph-mon[25851]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:16.626 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:16 smithi103 ceph-mon[25851]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-02-13T14:50:16.626 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:16 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:50:16.626 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:16 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.626 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:16 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.626 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:16 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.626 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:16 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:16.627 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:16 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:50:16.627 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:16 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:50:16.627 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:16 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:50:16.627 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:16 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.628 INFO:teuthology.orchestra.run.smithi103.stdout:55834574865 2024-02-13T14:50:16.628 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-13T14:50:16.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:16 smithi158 ceph-mon[27628]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:16.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:16 smithi158 ceph-mon[27628]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-02-13T14:50:16.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:16 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:50:16.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:16 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:16 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:16 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:16 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:16.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:16 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:50:16.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:16 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:50:16.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:16 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:50:16.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:16 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.816 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:16 smithi164 ceph-mon[27636]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:16.816 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:16 smithi164 ceph-mon[27636]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-02-13T14:50:16.816 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:16 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:50:16.816 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:16 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.817 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:16 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.817 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:16 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:16.817 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:16 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:16.817 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:16 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:50:16.817 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:16 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:50:16.817 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:16 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:50:16.817 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:16 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.672 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:17 smithi103 ceph-mon[25851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:50:17.672 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:17 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:50:17.672 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:17 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.672 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:17 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.672 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:17 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.672 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:17 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.672 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:17 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.672 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:17 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.673 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:17 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.673 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:17 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.673 INFO:teuthology.orchestra.run.smithi103.stdout:77309411340 2024-02-13T14:50:17.674 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-13T14:50:17.675 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:17 smithi158 ceph-mon[27628]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:50:17.675 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:17 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:50:17.675 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:17 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.675 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:17 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.676 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:17 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.676 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:17 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.676 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:17 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.676 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:17 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.676 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:17 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.676 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:17 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.810 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:17 smithi164 ceph-mon[27636]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:50:17.810 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:17 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi164", "caps": []}]: dispatch 2024-02-13T14:50:17.810 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:17 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.810 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:17 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.810 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:17 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.810 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:17 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.810 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:17 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.810 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:17 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.810 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:17 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:17.810 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:17 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:18.793 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:18 smithi164 ceph-mon[27636]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-02-13T14:50:18.793 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:18 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:18.793 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:18 smithi164 ceph-mon[27636]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-02-13T14:50:18.793 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:18 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:18.793 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:18 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:18.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:18 smithi103 ceph-mon[25851]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-02-13T14:50:18.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:18 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:18.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:18 smithi103 ceph-mon[25851]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-02-13T14:50:18.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:18 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:18.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:18 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:18.895 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:18 smithi158 ceph-mon[27628]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-02-13T14:50:18.895 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:18 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:18.895 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:18 smithi158 ceph-mon[27628]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-02-13T14:50:18.895 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:18 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:18.896 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:18 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.059 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:19.059 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:19.501 INFO:teuthology.orchestra.run.smithi103.stdout:34359738389 2024-02-13T14:50:19.501 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-13T14:50:19.725 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.725 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.725 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.725 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.725 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:19 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.785 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.785 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.785 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.785 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.785 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:19 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:50:19.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:50:19.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:50:19.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:19.930 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:19 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:20.708 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:20 smithi164 ceph-mon[27636]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-13T14:50:20.709 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:20 smithi164 ceph-mon[27636]: Deploying daemon alertmanager.a on smithi103 2024-02-13T14:50:20.884 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:20 smithi103 ceph-mon[25851]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-13T14:50:20.884 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:20 smithi103 ceph-mon[25851]: Deploying daemon alertmanager.a on smithi103 2024-02-13T14:50:20.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:20 smithi158 ceph-mon[27628]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-13T14:50:20.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:20 smithi158 ceph-mon[27628]: Deploying daemon alertmanager.a on smithi103 2024-02-13T14:50:22.037 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:22.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:22 smithi164 ceph-mon[27636]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-13T14:50:22.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:22 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:22 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:22 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:22 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:22 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:22 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:22 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:22 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:22 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:22 smithi103 ceph-mon[25851]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-13T14:50:22.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:22 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:22 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:22 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:22 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:22 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:22 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:22 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:22 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:22 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:22 smithi158 ceph-mon[27628]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-13T14:50:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:22 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:22 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:22 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:22 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:22 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:22 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:22 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:22 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:22.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:22 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:23.124 INFO:teuthology.orchestra.run.smithi103.stdout:77309411341 2024-02-13T14:50:23.351 INFO:teuthology.orchestra.run.smithi103.stdout:55834574866 2024-02-13T14:50:23.879 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:23 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/2208239595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T14:50:23.879 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:23 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/3819772171' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T14:50:23.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:23 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/2208239595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T14:50:23.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:23 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/3819772171' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T14:50:23.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:23 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/2208239595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-13T14:50:23.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:23 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/3819772171' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-13T14:50:24.847 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:24 smithi103 ceph-mon[25851]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T14:50:24.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:24 smithi164 ceph-mon[27636]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T14:50:24.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:24 smithi158 ceph-mon[27628]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-13T14:50:25.233 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411340 got 77309411341 for osd.2 2024-02-13T14:50:25.233 DEBUG:teuthology.parallel:result is None 2024-02-13T14:50:25.405 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:25 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ::ffff:172.21.15.158 - - [13/Feb/2024:14:50:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-13T14:50:25.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:25 smithi164 ceph-mon[27636]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T14:50:25.904 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:25 smithi103 ceph-mon[25851]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T14:50:25.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:25 smithi158 ceph-mon[27628]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-13T14:50:26.727 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574865 got 55834574866 for osd.1 2024-02-13T14:50:26.727 DEBUG:teuthology.parallel:result is None 2024-02-13T14:50:28.032 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:27 smithi103 ceph-mon[25851]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-13T14:50:28.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:27 smithi164 ceph-mon[27636]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-13T14:50:28.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:27 smithi158 ceph-mon[27628]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-13T14:50:28.784 INFO:teuthology.orchestra.run.smithi103.stdout:34359738391 2024-02-13T14:50:29.025 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:28 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:29.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:28 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:29.180 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:28 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:29.595 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738389 got 34359738391 for osd.0 2024-02-13T14:50:29.596 DEBUG:teuthology.parallel:result is None 2024-02-13T14:50:29.596 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-13T14:50:29.596 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T14:50:29.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:29 smithi103 ceph-mon[25851]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:29.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:29 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/3799457156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T14:50:30.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:29 smithi164 ceph-mon[27636]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:30.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:29 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/3799457156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T14:50:30.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:29 smithi158 ceph-mon[27628]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:30.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:29 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/3799457156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-13T14:50:31.231 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:31.814 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:31 smithi103 ceph-mon[25851]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:32.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:31 smithi164 ceph-mon[27636]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:32.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:31 smithi158 ceph-mon[27628]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:32.998 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:32 smithi103 podman[48882]: 2024-02-13 14:50:32.883205173 +0000 UTC m=+0.017587589 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-13T14:50:32.998 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:32 smithi103 podman[48882]: 2024-02-13 14:50:32.996195412 +0000 UTC m=+0.130577826 volume create 302107df868a9e76fee4c6fb45f1d0257764db5cd21636966a49c6974710e4e6 2024-02-13T14:50:33.073 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:50:33.075 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-02-13T14:50:33.260 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 podman[48882]: 2024-02-13 14:50:33.071512904 +0000 UTC m=+0.205895318 container create 23bbc8342d9c4311e1f2e9898d2f0303ec052b814499ba647315e6c95b770791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:50:33.846 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-mon[25851]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:33.847 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-mon[25851]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:50:33.847 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:33.847 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:33.847 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:33.847 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:33.847 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 podman[48882]: 2024-02-13 14:50:33.575421314 +0000 UTC m=+0.709803731 container init 23bbc8342d9c4311e1f2e9898d2f0303ec052b814499ba647315e6c95b770791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:50:33.847 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 podman[48882]: 2024-02-13 14:50:33.580922723 +0000 UTC m=+0.715305137 container start 23bbc8342d9c4311e1f2e9898d2f0303ec052b814499ba647315e6c95b770791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:50:33.847 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[48910]: ts=2024-02-13T14:50:33.593Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-13T14:50:33.847 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[48910]: ts=2024-02-13T14:50:33.593Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-13T14:50:33.848 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[48910]: ts=2024-02-13T14:50:33.594Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.103 port=9094 2024-02-13T14:50:33.848 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[48910]: ts=2024-02-13T14:50:33.595Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-13T14:50:33.848 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[48910]: ts=2024-02-13T14:50:33.617Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T14:50:33.848 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[48910]: ts=2024-02-13T14:50:33.617Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T14:50:33.848 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[48910]: ts=2024-02-13T14:50:33.620Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-13T14:50:33.848 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[48910]: ts=2024-02-13T14:50:33.620Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-13T14:50:33.848 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 bash[48882]: 23bbc8342d9c4311e1f2e9898d2f0303ec052b814499ba647315e6c95b770791 2024-02-13T14:50:33.848 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:33 smithi103 systemd[1]: Started Ceph alertmanager.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:50:34.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:33 smithi164 ceph-mon[27636]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:34.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:33 smithi164 ceph-mon[27636]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:50:34.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:33 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:34.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:33 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:34.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:33 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:34.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:33 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:34.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:50:33] ENGINE Bus STOPPING 2024-02-13T14:50:34.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:50:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T14:50:34.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:50:33] ENGINE Bus STOPPED 2024-02-13T14:50:34.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:33 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:50:33] ENGINE Bus STARTING 2024-02-13T14:50:34.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:50:34] ENGINE Serving on http://:::9283 2024-02-13T14:50:34.155 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:34 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:50:34] ENGINE Bus STARTED 2024-02-13T14:50:34.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:33 smithi158 ceph-mon[27628]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:34.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:33 smithi158 ceph-mon[27628]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:50:34.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:33 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:34.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:33 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:34.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:33 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:34.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:33 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:34.322 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-02-13T14:50:32.720899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96616,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126552,"statfs":{"total":287972524032,"available":287873589248,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002918"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707944+0000","last_change":"2024-02-13T14:49:50.031906+0000","last_active":"2024-02-13T14:50:12.707944+0000","last_peered":"2024-02-13T14:50:12.707944+0000","last_clean":"2024-02-13T14:50:12.707944+0000","last_became_active":"2024-02-13T14:49:50.030827+0000","last_became_peered":"2024-02-13T14:49:50.030827+0000","last_unstale":"2024-02-13T14:50:12.707944+0000","last_undegraded":"2024-02-13T14:50:12.707944+0000","last_fullsized":"2024-02-13T14:50:12.707944+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T17:10:05.176201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709479+0000","last_change":"2024-02-13T14:49:48.016438+0000","last_active":"2024-02-13T14:50:12.709479+0000","last_peered":"2024-02-13T14:50:12.709479+0000","last_clean":"2024-02-13T14:50:12.709479+0000","last_became_active":"2024-02-13T14:49:48.015194+0000","last_became_peered":"2024-02-13T14:49:48.015194+0000","last_unstale":"2024-02-13T14:50:12.709479+0000","last_undegraded":"2024-02-13T14:50:12.709479+0000","last_fullsized":"2024-02-13T14:50:12.709479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T17:10:29.485231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710421+0000","last_change":"2024-02-13T14:49:43.988493+0000","last_active":"2024-02-13T14:50:12.710421+0000","last_peered":"2024-02-13T14:50:12.710421+0000","last_clean":"2024-02-13T14:50:12.710421+0000","last_became_active":"2024-02-13T14:49:43.988114+0000","last_became_peered":"2024-02-13T14:49:43.988114+0000","last_unstale":"2024-02-13T14:50:12.710421+0000","last_undegraded":"2024-02-13T14:50:12.710421+0000","last_fullsized":"2024-02-13T14:50:12.710421+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:16:06.727488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709577+0000","last_change":"2024-02-13T14:49:46.001257+0000","last_active":"2024-02-13T14:50:12.709577+0000","last_peered":"2024-02-13T14:50:12.709577+0000","last_clean":"2024-02-13T14:50:12.709577+0000","last_became_active":"2024-02-13T14:49:46.000993+0000","last_became_peered":"2024-02-13T14:49:46.000993+0000","last_unstale":"2024-02-13T14:50:12.709577+0000","last_undegraded":"2024-02-13T14:50:12.709577+0000","last_fullsized":"2024-02-13T14:50:12.709577+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:33:41.628988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706714+0000","last_change":"2024-02-13T14:49:50.034415+0000","last_active":"2024-02-13T14:50:12.706714+0000","last_peered":"2024-02-13T14:50:12.706714+0000","last_clean":"2024-02-13T14:50:12.706714+0000","last_became_active":"2024-02-13T14:49:50.033995+0000","last_became_peered":"2024-02-13T14:49:50.033995+0000","last_unstale":"2024-02-13T14:50:12.706714+0000","last_undegraded":"2024-02-13T14:50:12.706714+0000","last_fullsized":"2024-02-13T14:50:12.706714+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T17:50:25.566494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707639+0000","last_change":"2024-02-13T14:49:48.020000+0000","last_active":"2024-02-13T14:50:12.707639+0000","last_peered":"2024-02-13T14:50:12.707639+0000","last_clean":"2024-02-13T14:50:12.707639+0000","last_became_active":"2024-02-13T14:49:48.019661+0000","last_became_peered":"2024-02-13T14:49:48.019661+0000","last_unstale":"2024-02-13T14:50:12.707639+0000","last_undegraded":"2024-02-13T14:50:12.707639+0000","last_fullsized":"2024-02-13T14:50:12.707639+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:55:12.042548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708969+0000","last_change":"2024-02-13T14:49:43.991078+0000","last_active":"2024-02-13T14:50:12.708969+0000","last_peered":"2024-02-13T14:50:12.708969+0000","last_clean":"2024-02-13T14:50:12.708969+0000","last_became_active":"2024-02-13T14:49:43.990692+0000","last_became_peered":"2024-02-13T14:49:43.990692+0000","last_unstale":"2024-02-13T14:50:12.708969+0000","last_undegraded":"2024-02-13T14:50:12.708969+0000","last_fullsized":"2024-02-13T14:50:12.708969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T17:55:28.171658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710091+0000","last_change":"2024-02-13T14:49:46.000738+0000","last_active":"2024-02-13T14:50:12.710091+0000","last_peered":"2024-02-13T14:50:12.710091+0000","last_clean":"2024-02-13T14:50:12.710091+0000","last_became_active":"2024-02-13T14:49:46.000360+0000","last_became_peered":"2024-02-13T14:49:46.000360+0000","last_unstale":"2024-02-13T14:50:12.710091+0000","last_undegraded":"2024-02-13T14:50:12.710091+0000","last_fullsized":"2024-02-13T14:50:12.710091+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:33:04.897697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709660+0000","last_change":"2024-02-13T14:49:50.031412+0000","last_active":"2024-02-13T14:50:12.709660+0000","last_peered":"2024-02-13T14:50:12.709660+0000","last_clean":"2024-02-13T14:50:12.709660+0000","last_became_active":"2024-02-13T14:49:50.031126+0000","last_became_peered":"2024-02-13T14:49:50.031126+0000","last_unstale":"2024-02-13T14:50:12.709660+0000","last_undegraded":"2024-02-13T14:50:12.709660+0000","last_fullsized":"2024-02-13T14:50:12.709660+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:06:41.356194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708238+0000","last_change":"2024-02-13T14:49:48.020280+0000","last_active":"2024-02-13T14:50:12.708238+0000","last_peered":"2024-02-13T14:50:12.708238+0000","last_clean":"2024-02-13T14:50:12.708238+0000","last_became_active":"2024-02-13T14:49:48.019919+0000","last_became_peered":"2024-02-13T14:49:48.019919+0000","last_unstale":"2024-02-13T14:50:12.708238+0000","last_undegraded":"2024-02-13T14:50:12.708238+0000","last_fullsized":"2024-02-13T14:50:12.708238+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:02:38.838333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713191+0000","last_change":"2024-02-13T14:49:43.984198+0000","last_active":"2024-02-13T14:50:12.713191+0000","last_peered":"2024-02-13T14:50:12.713191+0000","last_clean":"2024-02-13T14:50:12.713191+0000","last_became_active":"2024-02-13T14:49:43.983795+0000","last_became_peered":"2024-02-13T14:49:43.983795+0000","last_unstale":"2024-02-13T14:50:12.713191+0000","last_undegraded":"2024-02-13T14:50:12.713191+0000","last_fullsized":"2024-02-13T14:50:12.713191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:52:10.700900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710587+0000","last_change":"2024-02-13T14:49:46.001922+0000","last_active":"2024-02-13T14:50:12.710587+0000","last_peered":"2024-02-13T14:50:12.710587+0000","last_clean":"2024-02-13T14:50:12.710587+0000","last_became_active":"2024-02-13T14:49:46.001292+0000","last_became_peered":"2024-02-13T14:49:46.001292+0000","last_unstale":"2024-02-13T14:50:12.710587+0000","last_undegraded":"2024-02-13T14:50:12.710587+0000","last_fullsized":"2024-02-13T14:50:12.710587+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:21:25.941312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708069+0000","last_change":"2024-02-13T14:49:50.034238+0000","last_active":"2024-02-13T14:50:12.708069+0000","last_peered":"2024-02-13T14:50:12.708069+0000","last_clean":"2024-02-13T14:50:12.708069+0000","last_became_active":"2024-02-13T14:49:50.033869+0000","last_became_peered":"2024-02-13T14:49:50.033869+0000","last_unstale":"2024-02-13T14:50:12.708069+0000","last_undegraded":"2024-02-13T14:50:12.708069+0000","last_fullsized":"2024-02-13T14:50:12.708069+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:24:07.921260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708634+0000","last_change":"2024-02-13T14:49:48.019445+0000","last_active":"2024-02-13T14:50:12.708634+0000","last_peered":"2024-02-13T14:50:12.708634+0000","last_clean":"2024-02-13T14:50:12.708634+0000","last_became_active":"2024-02-13T14:49:48.019086+0000","last_became_peered":"2024-02-13T14:49:48.019086+0000","last_unstale":"2024-02-13T14:50:12.708634+0000","last_undegraded":"2024-02-13T14:50:12.708634+0000","last_fullsized":"2024-02-13T14:50:12.708634+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:22:43.864236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711095+0000","last_change":"2024-02-13T14:49:43.987991+0000","last_active":"2024-02-13T14:50:12.711095+0000","last_peered":"2024-02-13T14:50:12.711095+0000","last_clean":"2024-02-13T14:50:12.711095+0000","last_became_active":"2024-02-13T14:49:43.987676+0000","last_became_peered":"2024-02-13T14:49:43.987676+0000","last_unstale":"2024-02-13T14:50:12.711095+0000","last_undegraded":"2024-02-13T14:50:12.711095+0000","last_fullsized":"2024-02-13T14:50:12.711095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:59:41.234650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711383+0000","last_change":"2024-02-13T14:49:46.004198+0000","last_active":"2024-02-13T14:50:12.711383+0000","last_peered":"2024-02-13T14:50:12.711383+0000","last_clean":"2024-02-13T14:50:12.711383+0000","last_became_active":"2024-02-13T14:49:46.003888+0000","last_became_peered":"2024-02-13T14:49:46.003888+0000","last_unstale":"2024-02-13T14:50:12.711383+0000","last_undegraded":"2024-02-13T14:50:12.711383+0000","last_fullsized":"2024-02-13T14:50:12.711383+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:43:33.521621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709083+0000","last_change":"2024-02-13T14:49:50.031069+0000","last_active":"2024-02-13T14:50:12.709083+0000","last_peered":"2024-02-13T14:50:12.709083+0000","last_clean":"2024-02-13T14:50:12.709083+0000","last_became_active":"2024-02-13T14:49:50.030646+0000","last_became_peered":"2024-02-13T14:49:50.030646+0000","last_unstale":"2024-02-13T14:50:12.709083+0000","last_undegraded":"2024-02-13T14:50:12.709083+0000","last_fullsized":"2024-02-13T14:50:12.709083+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:00:53.592463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710215+0000","last_change":"2024-02-13T14:49:48.017156+0000","last_active":"2024-02-13T14:50:12.710215+0000","last_peered":"2024-02-13T14:50:12.710215+0000","last_clean":"2024-02-13T14:50:12.710215+0000","last_became_active":"2024-02-13T14:49:48.016677+0000","last_became_peered":"2024-02-13T14:49:48.016677+0000","last_unstale":"2024-02-13T14:50:12.710215+0000","last_undegraded":"2024-02-13T14:50:12.710215+0000","last_fullsized":"2024-02-13T14:50:12.710215+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:24:28.411282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713258+0000","last_change":"2024-02-13T14:49:43.984278+0000","last_active":"2024-02-13T14:50:12.713258+0000","last_peered":"2024-02-13T14:50:12.713258+0000","last_clean":"2024-02-13T14:50:12.713258+0000","last_became_active":"2024-02-13T14:49:43.983827+0000","last_became_peered":"2024-02-13T14:49:43.983827+0000","last_unstale":"2024-02-13T14:50:12.713258+0000","last_undegraded":"2024-02-13T14:50:12.713258+0000","last_fullsized":"2024-02-13T14:50:12.713258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:27:08.813333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711375+0000","last_change":"2024-02-13T14:49:46.004349+0000","last_active":"2024-02-13T14:50:12.711375+0000","last_peered":"2024-02-13T14:50:12.711375+0000","last_clean":"2024-02-13T14:50:12.711375+0000","last_became_active":"2024-02-13T14:49:46.004092+0000","last_became_peered":"2024-02-13T14:49:46.004092+0000","last_unstale":"2024-02-13T14:50:12.711375+0000","last_undegraded":"2024-02-13T14:50:12.711375+0000","last_fullsized":"2024-02-13T14:50:12.711375+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:00:51.594940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707856+0000","last_change":"2024-02-13T14:49:50.033339+0000","last_active":"2024-02-13T14:50:12.707856+0000","last_peered":"2024-02-13T14:50:12.707856+0000","last_clean":"2024-02-13T14:50:12.707856+0000","last_became_active":"2024-02-13T14:49:50.032842+0000","last_became_peered":"2024-02-13T14:49:50.032842+0000","last_unstale":"2024-02-13T14:50:12.707856+0000","last_undegraded":"2024-02-13T14:50:12.707856+0000","last_fullsized":"2024-02-13T14:50:12.707856+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:52:44.800408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710909+0000","last_change":"2024-02-13T14:49:48.016403+0000","last_active":"2024-02-13T14:50:12.710909+0000","last_peered":"2024-02-13T14:50:12.710909+0000","last_clean":"2024-02-13T14:50:12.710909+0000","last_became_active":"2024-02-13T14:49:48.015949+0000","last_became_peered":"2024-02-13T14:49:48.015949+0000","last_unstale":"2024-02-13T14:50:12.710909+0000","last_undegraded":"2024-02-13T14:50:12.710909+0000","last_fullsized":"2024-02-13T14:50:12.710909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:15:17.531499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708248+0000","last_change":"2024-02-13T14:49:43.984659+0000","last_active":"2024-02-13T14:50:12.708248+0000","last_peered":"2024-02-13T14:50:12.708248+0000","last_clean":"2024-02-13T14:50:12.708248+0000","last_became_active":"2024-02-13T14:49:43.983925+0000","last_became_peered":"2024-02-13T14:49:43.983925+0000","last_unstale":"2024-02-13T14:50:12.708248+0000","last_undegraded":"2024-02-13T14:50:12.708248+0000","last_fullsized":"2024-02-13T14:50:12.708248+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:01:09.481219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713082+0000","last_change":"2024-02-13T14:49:46.001887+0000","last_active":"2024-02-13T14:50:12.713082+0000","last_peered":"2024-02-13T14:50:12.713082+0000","last_clean":"2024-02-13T14:50:12.713082+0000","last_became_active":"2024-02-13T14:49:46.001598+0000","last_became_peered":"2024-02-13T14:49:46.001598+0000","last_unstale":"2024-02-13T14:50:12.713082+0000","last_undegraded":"2024-02-13T14:50:12.713082+0000","last_fullsized":"2024-02-13T14:50:12.713082+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:44:34.051990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707625+0000","last_change":"2024-02-13T14:49:50.033762+0000","last_active":"2024-02-13T14:50:12.707625+0000","last_peered":"2024-02-13T14:50:12.707625+0000","last_clean":"2024-02-13T14:50:12.707625+0000","last_became_active":"2024-02-13T14:49:50.033372+0000","last_became_peered":"2024-02-13T14:49:50.033372+0000","last_unstale":"2024-02-13T14:50:12.707625+0000","last_undegraded":"2024-02-13T14:50:12.707625+0000","last_fullsized":"2024-02-13T14:50:12.707625+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:50:34.269216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:19.037022+0000","last_change":"2024-02-13T14:49:48.017951+0000","last_active":"2024-02-13T14:50:19.037022+0000","last_peered":"2024-02-13T14:50:19.037022+0000","last_clean":"2024-02-13T14:50:19.037022+0000","last_became_active":"2024-02-13T14:49:48.016843+0000","last_became_peered":"2024-02-13T14:49:48.016843+0000","last_unstale":"2024-02-13T14:50:19.037022+0000","last_undegraded":"2024-02-13T14:50:19.037022+0000","last_fullsized":"2024-02-13T14:50:19.037022+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:47:27.407906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706833+0000","last_change":"2024-02-13T14:49:43.987751+0000","last_active":"2024-02-13T14:50:12.706833+0000","last_peered":"2024-02-13T14:50:12.706833+0000","last_clean":"2024-02-13T14:50:12.706833+0000","last_became_active":"2024-02-13T14:49:43.987160+0000","last_became_peered":"2024-02-13T14:49:43.987160+0000","last_unstale":"2024-02-13T14:50:12.706833+0000","last_undegraded":"2024-02-13T14:50:12.706833+0000","last_fullsized":"2024-02-13T14:50:12.706833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:50:19.697796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706557+0000","last_change":"2024-02-13T14:49:46.001845+0000","last_active":"2024-02-13T14:50:12.706557+0000","last_peered":"2024-02-13T14:50:12.706557+0000","last_clean":"2024-02-13T14:50:12.706557+0000","last_became_active":"2024-02-13T14:49:46.000969+0000","last_became_peered":"2024-02-13T14:49:46.000969+0000","last_unstale":"2024-02-13T14:50:12.706557+0000","last_undegraded":"2024-02-13T14:50:12.706557+0000","last_fullsized":"2024-02-13T14:50:12.706557+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:31:39.856295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710449+0000","last_change":"2024-02-13T14:49:50.032319+0000","last_active":"2024-02-13T14:50:12.710449+0000","last_peered":"2024-02-13T14:50:12.710449+0000","last_clean":"2024-02-13T14:50:12.710449+0000","last_became_active":"2024-02-13T14:49:50.031860+0000","last_became_peered":"2024-02-13T14:49:50.031860+0000","last_unstale":"2024-02-13T14:50:12.710449+0000","last_undegraded":"2024-02-13T14:50:12.710449+0000","last_fullsized":"2024-02-13T14:50:12.710449+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:43:02.044782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707924+0000","last_change":"2024-02-13T14:49:48.018584+0000","last_active":"2024-02-13T14:50:12.707924+0000","last_peered":"2024-02-13T14:50:12.707924+0000","last_clean":"2024-02-13T14:50:12.707924+0000","last_became_active":"2024-02-13T14:49:48.017483+0000","last_became_peered":"2024-02-13T14:49:48.017483+0000","last_unstale":"2024-02-13T14:50:12.707924+0000","last_undegraded":"2024-02-13T14:50:12.707924+0000","last_fullsized":"2024-02-13T14:50:12.707924+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:01:16.753767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709425+0000","last_change":"2024-02-13T14:49:43.984699+0000","last_active":"2024-02-13T14:50:12.709425+0000","last_peered":"2024-02-13T14:50:12.709425+0000","last_clean":"2024-02-13T14:50:12.709425+0000","last_became_active":"2024-02-13T14:49:43.984415+0000","last_became_peered":"2024-02-13T14:49:43.984415+0000","last_unstale":"2024-02-13T14:50:12.709425+0000","last_undegraded":"2024-02-13T14:50:12.709425+0000","last_fullsized":"2024-02-13T14:50:12.709425+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:17:05.997498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706583+0000","last_change":"2024-02-13T14:49:45.999543+0000","last_active":"2024-02-13T14:50:12.706583+0000","last_peered":"2024-02-13T14:50:12.706583+0000","last_clean":"2024-02-13T14:50:12.706583+0000","last_became_active":"2024-02-13T14:49:45.998762+0000","last_became_peered":"2024-02-13T14:49:45.998762+0000","last_unstale":"2024-02-13T14:50:12.706583+0000","last_undegraded":"2024-02-13T14:50:12.706583+0000","last_fullsized":"2024-02-13T14:50:12.706583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:19:13.130882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707824+0000","last_change":"2024-02-13T14:49:50.035304+0000","last_active":"2024-02-13T14:50:12.707824+0000","last_peered":"2024-02-13T14:50:12.707824+0000","last_clean":"2024-02-13T14:50:12.707824+0000","last_became_active":"2024-02-13T14:49:50.034994+0000","last_became_peered":"2024-02-13T14:49:50.034994+0000","last_unstale":"2024-02-13T14:50:12.707824+0000","last_undegraded":"2024-02-13T14:50:12.707824+0000","last_fullsized":"2024-02-13T14:50:12.707824+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:32:26.322389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706485+0000","last_change":"2024-02-13T14:49:48.016645+0000","last_active":"2024-02-13T14:50:12.706485+0000","last_peered":"2024-02-13T14:50:12.706485+0000","last_clean":"2024-02-13T14:50:12.706485+0000","last_became_active":"2024-02-13T14:49:48.016091+0000","last_became_peered":"2024-02-13T14:49:48.016091+0000","last_unstale":"2024-02-13T14:50:12.706485+0000","last_undegraded":"2024-02-13T14:50:12.706485+0000","last_fullsized":"2024-02-13T14:50:12.706485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:56:06.773843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711001+0000","last_change":"2024-02-13T14:49:43.986023+0000","last_active":"2024-02-13T14:50:12.711001+0000","last_peered":"2024-02-13T14:50:12.711001+0000","last_clean":"2024-02-13T14:50:12.711001+0000","last_became_active":"2024-02-13T14:49:43.985584+0000","last_became_peered":"2024-02-13T14:49:43.985584+0000","last_unstale":"2024-02-13T14:50:12.711001+0000","last_undegraded":"2024-02-13T14:50:12.711001+0000","last_fullsized":"2024-02-13T14:50:12.711001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:57:34.370906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710463+0000","last_change":"2024-02-13T14:49:46.003146+0000","last_active":"2024-02-13T14:50:12.710463+0000","last_peered":"2024-02-13T14:50:12.710463+0000","last_clean":"2024-02-13T14:50:12.710463+0000","last_became_active":"2024-02-13T14:49:46.002683+0000","last_became_peered":"2024-02-13T14:49:46.002683+0000","last_unstale":"2024-02-13T14:50:12.710463+0000","last_undegraded":"2024-02-13T14:50:12.710463+0000","last_fullsized":"2024-02-13T14:50:12.710463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:09:12.685827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707896+0000","last_change":"2024-02-13T14:49:50.031110+0000","last_active":"2024-02-13T14:50:12.707896+0000","last_peered":"2024-02-13T14:50:12.707896+0000","last_clean":"2024-02-13T14:50:12.707896+0000","last_became_active":"2024-02-13T14:49:50.030633+0000","last_became_peered":"2024-02-13T14:49:50.030633+0000","last_unstale":"2024-02-13T14:50:12.707896+0000","last_undegraded":"2024-02-13T14:50:12.707896+0000","last_fullsized":"2024-02-13T14:50:12.707896+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T14:50:44.378879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707006+0000","last_change":"2024-02-13T14:49:48.016354+0000","last_active":"2024-02-13T14:50:12.707006+0000","last_peered":"2024-02-13T14:50:12.707006+0000","last_clean":"2024-02-13T14:50:12.707006+0000","last_became_active":"2024-02-13T14:49:48.015977+0000","last_became_peered":"2024-02-13T14:49:48.015977+0000","last_unstale":"2024-02-13T14:50:12.707006+0000","last_undegraded":"2024-02-13T14:50:12.707006+0000","last_fullsized":"2024-02-13T14:50:12.707006+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:01:52.382248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709193+0000","last_change":"2024-02-13T14:49:43.984234+0000","last_active":"2024-02-13T14:50:12.709193+0000","last_peered":"2024-02-13T14:50:12.709193+0000","last_clean":"2024-02-13T14:50:12.709193+0000","last_became_active":"2024-02-13T14:49:43.983665+0000","last_became_peered":"2024-02-13T14:49:43.983665+0000","last_unstale":"2024-02-13T14:50:12.709193+0000","last_undegraded":"2024-02-13T14:50:12.709193+0000","last_fullsized":"2024-02-13T14:50:12.709193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:16:47.899383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710288+0000","last_change":"2024-02-13T14:49:46.003062+0000","last_active":"2024-02-13T14:50:12.710288+0000","last_peered":"2024-02-13T14:50:12.710288+0000","last_clean":"2024-02-13T14:50:12.710288+0000","last_became_active":"2024-02-13T14:49:46.002748+0000","last_became_peered":"2024-02-13T14:49:46.002748+0000","last_unstale":"2024-02-13T14:50:12.710288+0000","last_undegraded":"2024-02-13T14:50:12.710288+0000","last_fullsized":"2024-02-13T14:50:12.710288+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:57:33.201273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709909+0000","last_change":"2024-02-13T14:49:50.036030+0000","last_active":"2024-02-13T14:50:12.709909+0000","last_peered":"2024-02-13T14:50:12.709909+0000","last_clean":"2024-02-13T14:50:12.709909+0000","last_became_active":"2024-02-13T14:49:50.035721+0000","last_became_peered":"2024-02-13T14:49:50.035721+0000","last_unstale":"2024-02-13T14:50:12.709909+0000","last_undegraded":"2024-02-13T14:50:12.709909+0000","last_fullsized":"2024-02-13T14:50:12.709909+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:30:24.686800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708334+0000","last_change":"2024-02-13T14:49:48.019415+0000","last_active":"2024-02-13T14:50:12.708334+0000","last_peered":"2024-02-13T14:50:12.708334+0000","last_clean":"2024-02-13T14:50:12.708334+0000","last_became_active":"2024-02-13T14:49:48.019051+0000","last_became_peered":"2024-02-13T14:49:48.019051+0000","last_unstale":"2024-02-13T14:50:12.708334+0000","last_undegraded":"2024-02-13T14:50:12.708334+0000","last_fullsized":"2024-02-13T14:50:12.708334+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:52:02.151270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713276+0000","last_change":"2024-02-13T14:49:43.989768+0000","last_active":"2024-02-13T14:50:12.713276+0000","last_peered":"2024-02-13T14:50:12.713276+0000","last_clean":"2024-02-13T14:50:12.713276+0000","last_became_active":"2024-02-13T14:49:43.988724+0000","last_became_peered":"2024-02-13T14:49:43.988724+0000","last_unstale":"2024-02-13T14:50:12.713276+0000","last_undegraded":"2024-02-13T14:50:12.713276+0000","last_fullsized":"2024-02-13T14:50:12.713276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:37:16.142844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710755+0000","last_change":"2024-02-13T14:49:46.001807+0000","last_active":"2024-02-13T14:50:12.710755+0000","last_peered":"2024-02-13T14:50:12.710755+0000","last_clean":"2024-02-13T14:50:12.710755+0000","last_became_active":"2024-02-13T14:49:46.001317+0000","last_became_peered":"2024-02-13T14:49:46.001317+0000","last_unstale":"2024-02-13T14:50:12.710755+0000","last_undegraded":"2024-02-13T14:50:12.710755+0000","last_fullsized":"2024-02-13T14:50:12.710755+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:09:05.961764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709690+0000","last_change":"2024-02-13T14:49:50.035672+0000","last_active":"2024-02-13T14:50:12.709690+0000","last_peered":"2024-02-13T14:50:12.709690+0000","last_clean":"2024-02-13T14:50:12.709690+0000","last_became_active":"2024-02-13T14:49:50.035371+0000","last_became_peered":"2024-02-13T14:49:50.035371+0000","last_unstale":"2024-02-13T14:50:12.709690+0000","last_undegraded":"2024-02-13T14:50:12.709690+0000","last_fullsized":"2024-02-13T14:50:12.709690+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:22:15.419067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706775+0000","last_change":"2024-02-13T14:49:48.016954+0000","last_active":"2024-02-13T14:50:12.706775+0000","last_peered":"2024-02-13T14:50:12.706775+0000","last_clean":"2024-02-13T14:50:12.706775+0000","last_became_active":"2024-02-13T14:49:48.016611+0000","last_became_peered":"2024-02-13T14:49:48.016611+0000","last_unstale":"2024-02-13T14:50:12.706775+0000","last_undegraded":"2024-02-13T14:50:12.706775+0000","last_fullsized":"2024-02-13T14:50:12.706775+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:19:38.862691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710466+0000","last_change":"2024-02-13T14:49:43.985982+0000","last_active":"2024-02-13T14:50:12.710466+0000","last_peered":"2024-02-13T14:50:12.710466+0000","last_clean":"2024-02-13T14:50:12.710466+0000","last_became_active":"2024-02-13T14:49:43.985643+0000","last_became_peered":"2024-02-13T14:49:43.985643+0000","last_unstale":"2024-02-13T14:50:12.710466+0000","last_undegraded":"2024-02-13T14:50:12.710466+0000","last_fullsized":"2024-02-13T14:50:12.710466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:46:56.065502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711549+0000","last_change":"2024-02-13T14:49:46.002295+0000","last_active":"2024-02-13T14:50:12.711549+0000","last_peered":"2024-02-13T14:50:12.711549+0000","last_clean":"2024-02-13T14:50:12.711549+0000","last_became_active":"2024-02-13T14:49:46.001624+0000","last_became_peered":"2024-02-13T14:49:46.001624+0000","last_unstale":"2024-02-13T14:50:12.711549+0000","last_undegraded":"2024-02-13T14:50:12.711549+0000","last_fullsized":"2024-02-13T14:50:12.711549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:04:31.089130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708035+0000","last_change":"2024-02-13T14:49:50.034707+0000","last_active":"2024-02-13T14:50:12.708035+0000","last_peered":"2024-02-13T14:50:12.708035+0000","last_clean":"2024-02-13T14:50:12.708035+0000","last_became_active":"2024-02-13T14:49:50.034052+0000","last_became_peered":"2024-02-13T14:49:50.034052+0000","last_unstale":"2024-02-13T14:50:12.708035+0000","last_undegraded":"2024-02-13T14:50:12.708035+0000","last_fullsized":"2024-02-13T14:50:12.708035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:45:49.401829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:19.037020+0000","last_change":"2024-02-13T14:49:48.015793+0000","last_active":"2024-02-13T14:50:19.037020+0000","last_peered":"2024-02-13T14:50:19.037020+0000","last_clean":"2024-02-13T14:50:19.037020+0000","last_became_active":"2024-02-13T14:49:48.015494+0000","last_became_peered":"2024-02-13T14:49:48.015494+0000","last_unstale":"2024-02-13T14:50:19.037020+0000","last_undegraded":"2024-02-13T14:50:19.037020+0000","last_fullsized":"2024-02-13T14:50:19.037020+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:38:18.327521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713420+0000","last_change":"2024-02-13T14:49:43.988992+0000","last_active":"2024-02-13T14:50:12.713420+0000","last_peered":"2024-02-13T14:50:12.713420+0000","last_clean":"2024-02-13T14:50:12.713420+0000","last_became_active":"2024-02-13T14:49:43.988704+0000","last_became_peered":"2024-02-13T14:49:43.988704+0000","last_unstale":"2024-02-13T14:50:12.713420+0000","last_undegraded":"2024-02-13T14:50:12.713420+0000","last_fullsized":"2024-02-13T14:50:12.713420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:17:17.114027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707231+0000","last_change":"2024-02-13T14:49:46.001820+0000","last_active":"2024-02-13T14:50:12.707231+0000","last_peered":"2024-02-13T14:50:12.707231+0000","last_clean":"2024-02-13T14:50:12.707231+0000","last_became_active":"2024-02-13T14:49:46.001267+0000","last_became_peered":"2024-02-13T14:49:46.001267+0000","last_unstale":"2024-02-13T14:50:12.707231+0000","last_undegraded":"2024-02-13T14:50:12.707231+0000","last_fullsized":"2024-02-13T14:50:12.707231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:31:12.315257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708259+0000","last_change":"2024-02-13T14:49:50.034532+0000","last_active":"2024-02-13T14:50:12.708259+0000","last_peered":"2024-02-13T14:50:12.708259+0000","last_clean":"2024-02-13T14:50:12.708259+0000","last_became_active":"2024-02-13T14:49:50.034158+0000","last_became_peered":"2024-02-13T14:49:50.034158+0000","last_unstale":"2024-02-13T14:50:12.708259+0000","last_undegraded":"2024-02-13T14:50:12.708259+0000","last_fullsized":"2024-02-13T14:50:12.708259+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:27:01.275495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:19.037022+0000","last_change":"2024-02-13T14:49:48.016146+0000","last_active":"2024-02-13T14:50:19.037022+0000","last_peered":"2024-02-13T14:50:19.037022+0000","last_clean":"2024-02-13T14:50:19.037022+0000","last_became_active":"2024-02-13T14:49:48.015330+0000","last_became_peered":"2024-02-13T14:49:48.015330+0000","last_unstale":"2024-02-13T14:50:19.037022+0000","last_undegraded":"2024-02-13T14:50:19.037022+0000","last_fullsized":"2024-02-13T14:50:19.037022+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:27:54.322942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710908+0000","last_change":"2024-02-13T14:49:43.987875+0000","last_active":"2024-02-13T14:50:12.710908+0000","last_peered":"2024-02-13T14:50:12.710908+0000","last_clean":"2024-02-13T14:50:12.710908+0000","last_became_active":"2024-02-13T14:49:43.987560+0000","last_became_peered":"2024-02-13T14:49:43.987560+0000","last_unstale":"2024-02-13T14:50:12.710908+0000","last_undegraded":"2024-02-13T14:50:12.710908+0000","last_fullsized":"2024-02-13T14:50:12.710908+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:35:56.749045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708817+0000","last_change":"2024-02-13T14:49:46.001736+0000","last_active":"2024-02-13T14:50:12.708817+0000","last_peered":"2024-02-13T14:50:12.708817+0000","last_clean":"2024-02-13T14:50:12.708817+0000","last_became_active":"2024-02-13T14:49:46.001264+0000","last_became_peered":"2024-02-13T14:49:46.001264+0000","last_unstale":"2024-02-13T14:50:12.708817+0000","last_undegraded":"2024-02-13T14:50:12.708817+0000","last_fullsized":"2024-02-13T14:50:12.708817+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:30:11.936954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708724+0000","last_change":"2024-02-13T14:49:50.031258+0000","last_active":"2024-02-13T14:50:12.708724+0000","last_peered":"2024-02-13T14:50:12.708724+0000","last_clean":"2024-02-13T14:50:12.708724+0000","last_became_active":"2024-02-13T14:49:50.030786+0000","last_became_peered":"2024-02-13T14:49:50.030786+0000","last_unstale":"2024-02-13T14:50:12.708724+0000","last_undegraded":"2024-02-13T14:50:12.708724+0000","last_fullsized":"2024-02-13T14:50:12.708724+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:59:04.844751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709929+0000","last_change":"2024-02-13T14:49:48.019153+0000","last_active":"2024-02-13T14:50:12.709929+0000","last_peered":"2024-02-13T14:50:12.709929+0000","last_clean":"2024-02-13T14:50:12.709929+0000","last_became_active":"2024-02-13T14:49:48.018803+0000","last_became_peered":"2024-02-13T14:49:48.018803+0000","last_unstale":"2024-02-13T14:50:12.709929+0000","last_undegraded":"2024-02-13T14:50:12.709929+0000","last_fullsized":"2024-02-13T14:50:12.709929+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:43:54.058564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707758+0000","last_change":"2024-02-13T14:49:43.983981+0000","last_active":"2024-02-13T14:50:12.707758+0000","last_peered":"2024-02-13T14:50:12.707758+0000","last_clean":"2024-02-13T14:50:12.707758+0000","last_became_active":"2024-02-13T14:49:43.983571+0000","last_became_peered":"2024-02-13T14:49:43.983571+0000","last_unstale":"2024-02-13T14:50:12.707758+0000","last_undegraded":"2024-02-13T14:50:12.707758+0000","last_fullsized":"2024-02-13T14:50:12.707758+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:51:35.609959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707336+0000","last_change":"2024-02-13T14:49:46.002760+0000","last_active":"2024-02-13T14:50:12.707336+0000","last_peered":"2024-02-13T14:50:12.707336+0000","last_clean":"2024-02-13T14:50:12.707336+0000","last_became_active":"2024-02-13T14:49:46.002205+0000","last_became_peered":"2024-02-13T14:49:46.002205+0000","last_unstale":"2024-02-13T14:50:12.707336+0000","last_undegraded":"2024-02-13T14:50:12.707336+0000","last_fullsized":"2024-02-13T14:50:12.707336+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:30:02.807948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709686+0000","last_change":"2024-02-13T14:49:50.032823+0000","last_active":"2024-02-13T14:50:12.709686+0000","last_peered":"2024-02-13T14:50:12.709686+0000","last_clean":"2024-02-13T14:50:12.709686+0000","last_became_active":"2024-02-13T14:49:50.032454+0000","last_became_peered":"2024-02-13T14:49:50.032454+0000","last_unstale":"2024-02-13T14:50:12.709686+0000","last_undegraded":"2024-02-13T14:50:12.709686+0000","last_fullsized":"2024-02-13T14:50:12.709686+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:26:18.706216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708016+0000","last_change":"2024-02-13T14:49:48.018734+0000","last_active":"2024-02-13T14:50:12.708016+0000","last_peered":"2024-02-13T14:50:12.708016+0000","last_clean":"2024-02-13T14:50:12.708016+0000","last_became_active":"2024-02-13T14:49:48.018342+0000","last_became_peered":"2024-02-13T14:49:48.018342+0000","last_unstale":"2024-02-13T14:50:12.708016+0000","last_undegraded":"2024-02-13T14:50:12.708016+0000","last_fullsized":"2024-02-13T14:50:12.708016+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:33:41.799571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709533+0000","last_change":"2024-02-13T14:49:43.987748+0000","last_active":"2024-02-13T14:50:12.709533+0000","last_peered":"2024-02-13T14:50:12.709533+0000","last_clean":"2024-02-13T14:50:12.709533+0000","last_became_active":"2024-02-13T14:49:43.987463+0000","last_became_peered":"2024-02-13T14:49:43.987463+0000","last_unstale":"2024-02-13T14:50:12.709533+0000","last_undegraded":"2024-02-13T14:50:12.709533+0000","last_fullsized":"2024-02-13T14:50:12.709533+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:14:45.012311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710328+0000","last_change":"2024-02-13T14:49:46.002448+0000","last_active":"2024-02-13T14:50:12.710328+0000","last_peered":"2024-02-13T14:50:12.710328+0000","last_clean":"2024-02-13T14:50:12.710328+0000","last_became_active":"2024-02-13T14:49:46.002205+0000","last_became_peered":"2024-02-13T14:49:46.002205+0000","last_unstale":"2024-02-13T14:50:12.710328+0000","last_undegraded":"2024-02-13T14:50:12.710328+0000","last_fullsized":"2024-02-13T14:50:12.710328+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:17:11.020957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711181+0000","last_change":"2024-02-13T14:49:50.037297+0000","last_active":"2024-02-13T14:50:12.711181+0000","last_peered":"2024-02-13T14:50:12.711181+0000","last_clean":"2024-02-13T14:50:12.711181+0000","last_became_active":"2024-02-13T14:49:50.036878+0000","last_became_peered":"2024-02-13T14:49:50.036878+0000","last_unstale":"2024-02-13T14:50:12.711181+0000","last_undegraded":"2024-02-13T14:50:12.711181+0000","last_fullsized":"2024-02-13T14:50:12.711181+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:35:36.643689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713599+0000","last_change":"2024-02-13T14:49:43.984783+0000","last_active":"2024-02-13T14:50:12.713599+0000","last_peered":"2024-02-13T14:50:12.713599+0000","last_clean":"2024-02-13T14:50:12.713599+0000","last_became_active":"2024-02-13T14:49:43.984004+0000","last_became_peered":"2024-02-13T14:49:43.984004+0000","last_unstale":"2024-02-13T14:50:12.713599+0000","last_undegraded":"2024-02-13T14:50:12.713599+0000","last_fullsized":"2024-02-13T14:50:12.713599+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:57:33.524731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711204+0000","last_change":"2024-02-13T14:49:46.002699+0000","last_active":"2024-02-13T14:50:12.711204+0000","last_peered":"2024-02-13T14:50:12.711204+0000","last_clean":"2024-02-13T14:50:12.711204+0000","last_became_active":"2024-02-13T14:49:46.002401+0000","last_became_peered":"2024-02-13T14:49:46.002401+0000","last_unstale":"2024-02-13T14:50:12.711204+0000","last_undegraded":"2024-02-13T14:50:12.711204+0000","last_fullsized":"2024-02-13T14:50:12.711204+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:39:29.043303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:19.037096+0000","last_change":"2024-02-13T14:49:48.015846+0000","last_active":"2024-02-13T14:50:19.037096+0000","last_peered":"2024-02-13T14:50:19.037096+0000","last_clean":"2024-02-13T14:50:19.037096+0000","last_became_active":"2024-02-13T14:49:48.015286+0000","last_became_peered":"2024-02-13T14:49:48.015286+0000","last_unstale":"2024-02-13T14:50:19.037096+0000","last_undegraded":"2024-02-13T14:50:19.037096+0000","last_fullsized":"2024-02-13T14:50:19.037096+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:23:29.086268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707523+0000","last_change":"2024-02-13T14:49:50.028918+0000","last_active":"2024-02-13T14:50:12.707523+0000","last_peered":"2024-02-13T14:50:12.707523+0000","last_clean":"2024-02-13T14:50:12.707523+0000","last_became_active":"2024-02-13T14:49:50.028199+0000","last_became_peered":"2024-02-13T14:49:50.028199+0000","last_unstale":"2024-02-13T14:50:12.707523+0000","last_undegraded":"2024-02-13T14:50:12.707523+0000","last_fullsized":"2024-02-13T14:50:12.707523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:01:00.591300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709346+0000","last_change":"2024-02-13T14:49:43.988261+0000","last_active":"2024-02-13T14:50:12.709346+0000","last_peered":"2024-02-13T14:50:12.709346+0000","last_clean":"2024-02-13T14:50:12.709346+0000","last_became_active":"2024-02-13T14:49:43.987854+0000","last_became_peered":"2024-02-13T14:49:43.987854+0000","last_unstale":"2024-02-13T14:50:12.709346+0000","last_undegraded":"2024-02-13T14:50:12.709346+0000","last_fullsized":"2024-02-13T14:50:12.709346+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:48:44.471710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:19.037241+0000","last_change":"2024-02-13T14:49:46.002251+0000","last_active":"2024-02-13T14:50:19.037241+0000","last_peered":"2024-02-13T14:50:19.037241+0000","last_clean":"2024-02-13T14:50:19.037241+0000","last_became_active":"2024-02-13T14:49:46.001489+0000","last_became_peered":"2024-02-13T14:49:46.001489+0000","last_unstale":"2024-02-13T14:50:19.037241+0000","last_undegraded":"2024-02-13T14:50:19.037241+0000","last_fullsized":"2024-02-13T14:50:19.037241+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:30:11.548080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710154+0000","last_change":"2024-02-13T14:49:48.016152+0000","last_active":"2024-02-13T14:50:12.710154+0000","last_peered":"2024-02-13T14:50:12.710154+0000","last_clean":"2024-02-13T14:50:12.710154+0000","last_became_active":"2024-02-13T14:49:48.015597+0000","last_became_peered":"2024-02-13T14:49:48.015597+0000","last_unstale":"2024-02-13T14:50:12.710154+0000","last_undegraded":"2024-02-13T14:50:12.710154+0000","last_fullsized":"2024-02-13T14:50:12.710154+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:32:41.154762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708504+0000","last_change":"2024-02-13T14:49:50.033013+0000","last_active":"2024-02-13T14:50:12.708504+0000","last_peered":"2024-02-13T14:50:12.708504+0000","last_clean":"2024-02-13T14:50:12.708504+0000","last_became_active":"2024-02-13T14:49:50.032502+0000","last_became_peered":"2024-02-13T14:49:50.032502+0000","last_unstale":"2024-02-13T14:50:12.708504+0000","last_undegraded":"2024-02-13T14:50:12.708504+0000","last_fullsized":"2024-02-13T14:50:12.708504+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:44:59.486399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708428+0000","last_change":"2024-02-13T14:49:43.984911+0000","last_active":"2024-02-13T14:50:12.708428+0000","last_peered":"2024-02-13T14:50:12.708428+0000","last_clean":"2024-02-13T14:50:12.708428+0000","last_became_active":"2024-02-13T14:49:43.984530+0000","last_became_peered":"2024-02-13T14:49:43.984530+0000","last_unstale":"2024-02-13T14:50:12.708428+0000","last_undegraded":"2024-02-13T14:50:12.708428+0000","last_fullsized":"2024-02-13T14:50:12.708428+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:06:22.508793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708900+0000","last_change":"2024-02-13T14:49:46.002011+0000","last_active":"2024-02-13T14:50:12.708900+0000","last_peered":"2024-02-13T14:50:12.708900+0000","last_clean":"2024-02-13T14:50:12.708900+0000","last_became_active":"2024-02-13T14:49:46.001146+0000","last_became_peered":"2024-02-13T14:49:46.001146+0000","last_unstale":"2024-02-13T14:50:12.708900+0000","last_undegraded":"2024-02-13T14:50:12.708900+0000","last_fullsized":"2024-02-13T14:50:12.708900+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:58:44.212130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708802+0000","last_change":"2024-02-13T14:49:48.017518+0000","last_active":"2024-02-13T14:50:12.708802+0000","last_peered":"2024-02-13T14:50:12.708802+0000","last_clean":"2024-02-13T14:50:12.708802+0000","last_became_active":"2024-02-13T14:49:48.017207+0000","last_became_peered":"2024-02-13T14:49:48.017207+0000","last_unstale":"2024-02-13T14:50:12.708802+0000","last_undegraded":"2024-02-13T14:50:12.708802+0000","last_fullsized":"2024-02-13T14:50:12.708802+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:04:13.651245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708674+0000","last_change":"2024-02-13T14:49:50.034965+0000","last_active":"2024-02-13T14:50:12.708674+0000","last_peered":"2024-02-13T14:50:12.708674+0000","last_clean":"2024-02-13T14:50:12.708674+0000","last_became_active":"2024-02-13T14:49:50.034624+0000","last_became_peered":"2024-02-13T14:49:50.034624+0000","last_unstale":"2024-02-13T14:50:12.708674+0000","last_undegraded":"2024-02-13T14:50:12.708674+0000","last_fullsized":"2024-02-13T14:50:12.708674+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:06:08.558445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709558+0000","last_change":"2024-02-13T14:49:46.002350+0000","last_active":"2024-02-13T14:50:12.709558+0000","last_peered":"2024-02-13T14:50:12.709558+0000","last_clean":"2024-02-13T14:50:12.709558+0000","last_became_active":"2024-02-13T14:49:46.002053+0000","last_became_peered":"2024-02-13T14:49:46.002053+0000","last_unstale":"2024-02-13T14:50:12.709558+0000","last_undegraded":"2024-02-13T14:50:12.709558+0000","last_fullsized":"2024-02-13T14:50:12.709558+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:30:11.897592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706645+0000","last_change":"2024-02-13T14:49:43.987540+0000","last_active":"2024-02-13T14:50:12.706645+0000","last_peered":"2024-02-13T14:50:12.706645+0000","last_clean":"2024-02-13T14:50:12.706645+0000","last_became_active":"2024-02-13T14:49:43.987052+0000","last_became_peered":"2024-02-13T14:49:43.987052+0000","last_unstale":"2024-02-13T14:50:12.706645+0000","last_undegraded":"2024-02-13T14:50:12.706645+0000","last_fullsized":"2024-02-13T14:50:12.706645+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:38:56.196725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710206+0000","last_change":"2024-02-13T14:49:48.020513+0000","last_active":"2024-02-13T14:50:12.710206+0000","last_peered":"2024-02-13T14:50:12.710206+0000","last_clean":"2024-02-13T14:50:12.710206+0000","last_became_active":"2024-02-13T14:49:48.019602+0000","last_became_peered":"2024-02-13T14:49:48.019602+0000","last_unstale":"2024-02-13T14:50:12.710206+0000","last_undegraded":"2024-02-13T14:50:12.710206+0000","last_fullsized":"2024-02-13T14:50:12.710206+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:26:08.069433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709861+0000","last_change":"2024-02-13T14:49:50.037294+0000","last_active":"2024-02-13T14:50:12.709861+0000","last_peered":"2024-02-13T14:50:12.709861+0000","last_clean":"2024-02-13T14:50:12.709861+0000","last_became_active":"2024-02-13T14:49:50.036614+0000","last_became_peered":"2024-02-13T14:49:50.036614+0000","last_unstale":"2024-02-13T14:50:12.709861+0000","last_undegraded":"2024-02-13T14:50:12.709861+0000","last_fullsized":"2024-02-13T14:50:12.709861+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T17:57:00.459745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707050+0000","last_change":"2024-02-13T14:49:46.001762+0000","last_active":"2024-02-13T14:50:12.707050+0000","last_peered":"2024-02-13T14:50:12.707050+0000","last_clean":"2024-02-13T14:50:12.707050+0000","last_became_active":"2024-02-13T14:49:46.001221+0000","last_became_peered":"2024-02-13T14:49:46.001221+0000","last_unstale":"2024-02-13T14:50:12.707050+0000","last_undegraded":"2024-02-13T14:50:12.707050+0000","last_fullsized":"2024-02-13T14:50:12.707050+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:31:58.993425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708651+0000","last_change":"2024-02-13T14:49:43.982793+0000","last_active":"2024-02-13T14:50:12.708651+0000","last_peered":"2024-02-13T14:50:12.708651+0000","last_clean":"2024-02-13T14:50:12.708651+0000","last_became_active":"2024-02-13T14:49:43.982528+0000","last_became_peered":"2024-02-13T14:49:43.982528+0000","last_unstale":"2024-02-13T14:50:12.708651+0000","last_undegraded":"2024-02-13T14:50:12.708651+0000","last_fullsized":"2024-02-13T14:50:12.708651+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:47:38.808484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:14.808693+0000","last_change":"2024-02-13T14:49:30.308712+0000","last_active":"2024-02-13T14:50:14.808693+0000","last_peered":"2024-02-13T14:50:14.808693+0000","last_clean":"2024-02-13T14:50:14.808693+0000","last_became_active":"2024-02-13T14:49:30.308454+0000","last_became_peered":"2024-02-13T14:49:30.308454+0000","last_unstale":"2024-02-13T14:50:14.808693+0000","last_undegraded":"2024-02-13T14:50:14.808693+0000","last_fullsized":"2024-02-13T14:50:14.808693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:29.292511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:29.292511+0000","last_clean_scrub_stamp":"2024-02-13T14:49:29.292511+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:10:17.580074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708877+0000","last_change":"2024-02-13T14:49:48.020032+0000","last_active":"2024-02-13T14:50:12.708877+0000","last_peered":"2024-02-13T14:50:12.708877+0000","last_clean":"2024-02-13T14:50:12.708877+0000","last_became_active":"2024-02-13T14:49:48.019675+0000","last_became_peered":"2024-02-13T14:49:48.019675+0000","last_unstale":"2024-02-13T14:50:12.708877+0000","last_undegraded":"2024-02-13T14:50:12.708877+0000","last_fullsized":"2024-02-13T14:50:12.708877+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:45:43.794174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710459+0000","last_change":"2024-02-13T14:49:50.030107+0000","last_active":"2024-02-13T14:50:12.710459+0000","last_peered":"2024-02-13T14:50:12.710459+0000","last_clean":"2024-02-13T14:50:12.710459+0000","last_became_active":"2024-02-13T14:49:50.029703+0000","last_became_peered":"2024-02-13T14:49:50.029703+0000","last_unstale":"2024-02-13T14:50:12.710459+0000","last_undegraded":"2024-02-13T14:50:12.710459+0000","last_fullsized":"2024-02-13T14:50:12.710459+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:45:18.165088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706950+0000","last_change":"2024-02-13T14:49:45.999758+0000","last_active":"2024-02-13T14:50:12.706950+0000","last_peered":"2024-02-13T14:50:12.706950+0000","last_clean":"2024-02-13T14:50:12.706950+0000","last_became_active":"2024-02-13T14:49:45.999258+0000","last_became_peered":"2024-02-13T14:49:45.999258+0000","last_unstale":"2024-02-13T14:50:12.706950+0000","last_undegraded":"2024-02-13T14:50:12.706950+0000","last_fullsized":"2024-02-13T14:50:12.706950+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:53:52.999026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707415+0000","last_change":"2024-02-13T14:49:43.986159+0000","last_active":"2024-02-13T14:50:12.707415+0000","last_peered":"2024-02-13T14:50:12.707415+0000","last_clean":"2024-02-13T14:50:12.707415+0000","last_became_active":"2024-02-13T14:49:43.985652+0000","last_became_peered":"2024-02-13T14:49:43.985652+0000","last_unstale":"2024-02-13T14:50:12.707415+0000","last_undegraded":"2024-02-13T14:50:12.707415+0000","last_fullsized":"2024-02-13T14:50:12.707415+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:28:59.940724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710904+0000","last_change":"2024-02-13T14:49:48.016101+0000","last_active":"2024-02-13T14:50:12.710904+0000","last_peered":"2024-02-13T14:50:12.710904+0000","last_clean":"2024-02-13T14:50:12.710904+0000","last_became_active":"2024-02-13T14:49:48.015684+0000","last_became_peered":"2024-02-13T14:49:48.015684+0000","last_unstale":"2024-02-13T14:50:12.710904+0000","last_undegraded":"2024-02-13T14:50:12.710904+0000","last_fullsized":"2024-02-13T14:50:12.710904+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:05:14.129159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708138+0000","last_change":"2024-02-13T14:49:50.030492+0000","last_active":"2024-02-13T14:50:12.708138+0000","last_peered":"2024-02-13T14:50:12.708138+0000","last_clean":"2024-02-13T14:50:12.708138+0000","last_became_active":"2024-02-13T14:49:50.029776+0000","last_became_peered":"2024-02-13T14:49:50.029776+0000","last_unstale":"2024-02-13T14:50:12.708138+0000","last_undegraded":"2024-02-13T14:50:12.708138+0000","last_fullsized":"2024-02-13T14:50:12.708138+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:24:42.344672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713014+0000","last_change":"2024-02-13T14:49:43.989536+0000","last_active":"2024-02-13T14:50:12.713014+0000","last_peered":"2024-02-13T14:50:12.713014+0000","last_clean":"2024-02-13T14:50:12.713014+0000","last_became_active":"2024-02-13T14:49:43.988725+0000","last_became_peered":"2024-02-13T14:49:43.988725+0000","last_unstale":"2024-02-13T14:50:12.713014+0000","last_undegraded":"2024-02-13T14:50:12.713014+0000","last_fullsized":"2024-02-13T14:50:12.713014+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:21:02.435472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:19.036889+0000","last_change":"2024-02-13T14:49:46.000818+0000","last_active":"2024-02-13T14:50:19.036889+0000","last_peered":"2024-02-13T14:50:19.036889+0000","last_clean":"2024-02-13T14:50:19.036889+0000","last_became_active":"2024-02-13T14:49:46.000286+0000","last_became_peered":"2024-02-13T14:49:46.000286+0000","last_unstale":"2024-02-13T14:50:19.036889+0000","last_undegraded":"2024-02-13T14:50:19.036889+0000","last_fullsized":"2024-02-13T14:50:19.036889+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:11:56.998741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:19.036994+0000","last_change":"2024-02-13T14:49:48.015568+0000","last_active":"2024-02-13T14:50:19.036994+0000","last_peered":"2024-02-13T14:50:19.036994+0000","last_clean":"2024-02-13T14:50:19.036994+0000","last_became_active":"2024-02-13T14:49:48.015251+0000","last_became_peered":"2024-02-13T14:49:48.015251+0000","last_unstale":"2024-02-13T14:50:19.036994+0000","last_undegraded":"2024-02-13T14:50:19.036994+0000","last_fullsized":"2024-02-13T14:50:19.036994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:20:31.800839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707437+0000","last_change":"2024-02-13T14:49:50.032427+0000","last_active":"2024-02-13T14:50:12.707437+0000","last_peered":"2024-02-13T14:50:12.707437+0000","last_clean":"2024-02-13T14:50:12.707437+0000","last_became_active":"2024-02-13T14:49:50.031984+0000","last_became_peered":"2024-02-13T14:49:50.031984+0000","last_unstale":"2024-02-13T14:50:12.707437+0000","last_undegraded":"2024-02-13T14:50:12.707437+0000","last_fullsized":"2024-02-13T14:50:12.707437+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:29:36.818832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709136+0000","last_change":"2024-02-13T14:49:43.990930+0000","last_active":"2024-02-13T14:50:12.709136+0000","last_peered":"2024-02-13T14:50:12.709136+0000","last_clean":"2024-02-13T14:50:12.709136+0000","last_became_active":"2024-02-13T14:49:43.990386+0000","last_became_peered":"2024-02-13T14:49:43.990386+0000","last_unstale":"2024-02-13T14:50:12.709136+0000","last_undegraded":"2024-02-13T14:50:12.709136+0000","last_fullsized":"2024-02-13T14:50:12.709136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:28:33.949170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710545+0000","last_change":"2024-02-13T14:49:46.003030+0000","last_active":"2024-02-13T14:50:12.710545+0000","last_peered":"2024-02-13T14:50:12.710545+0000","last_clean":"2024-02-13T14:50:12.710545+0000","last_became_active":"2024-02-13T14:49:46.002730+0000","last_became_peered":"2024-02-13T14:49:46.002730+0000","last_unstale":"2024-02-13T14:50:12.710545+0000","last_undegraded":"2024-02-13T14:50:12.710545+0000","last_fullsized":"2024-02-13T14:50:12.710545+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:03:36.703793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:19.036895+0000","last_change":"2024-02-13T14:49:48.016665+0000","last_active":"2024-02-13T14:50:19.036895+0000","last_peered":"2024-02-13T14:50:19.036895+0000","last_clean":"2024-02-13T14:50:19.036895+0000","last_became_active":"2024-02-13T14:49:48.016336+0000","last_became_peered":"2024-02-13T14:49:48.016336+0000","last_unstale":"2024-02-13T14:50:19.036895+0000","last_undegraded":"2024-02-13T14:50:19.036895+0000","last_fullsized":"2024-02-13T14:50:19.036895+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:23:05.130567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711081+0000","last_change":"2024-02-13T14:49:50.037038+0000","last_active":"2024-02-13T14:50:12.711081+0000","last_peered":"2024-02-13T14:50:12.711081+0000","last_clean":"2024-02-13T14:50:12.711081+0000","last_became_active":"2024-02-13T14:49:50.036715+0000","last_became_peered":"2024-02-13T14:49:50.036715+0000","last_unstale":"2024-02-13T14:50:12.711081+0000","last_undegraded":"2024-02-13T14:50:12.711081+0000","last_fullsized":"2024-02-13T14:50:12.711081+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:00:49.715548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710837+0000","last_change":"2024-02-13T14:49:43.983241+0000","last_active":"2024-02-13T14:50:12.710837+0000","last_peered":"2024-02-13T14:50:12.710837+0000","last_clean":"2024-02-13T14:50:12.710837+0000","last_became_active":"2024-02-13T14:49:43.982897+0000","last_became_peered":"2024-02-13T14:49:43.982897+0000","last_unstale":"2024-02-13T14:50:12.710837+0000","last_undegraded":"2024-02-13T14:50:12.710837+0000","last_fullsized":"2024-02-13T14:50:12.710837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:00:51.552200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708542+0000","last_change":"2024-02-13T14:49:46.001842+0000","last_active":"2024-02-13T14:50:12.708542+0000","last_peered":"2024-02-13T14:50:12.708542+0000","last_clean":"2024-02-13T14:50:12.708542+0000","last_became_active":"2024-02-13T14:49:46.001528+0000","last_became_peered":"2024-02-13T14:49:46.001528+0000","last_unstale":"2024-02-13T14:50:12.708542+0000","last_undegraded":"2024-02-13T14:50:12.708542+0000","last_fullsized":"2024-02-13T14:50:12.708542+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:02:52.234290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709583+0000","last_change":"2024-02-13T14:49:48.016738+0000","last_active":"2024-02-13T14:50:12.709583+0000","last_peered":"2024-02-13T14:50:12.709583+0000","last_clean":"2024-02-13T14:50:12.709583+0000","last_became_active":"2024-02-13T14:49:48.016423+0000","last_became_peered":"2024-02-13T14:49:48.016423+0000","last_unstale":"2024-02-13T14:50:12.709583+0000","last_undegraded":"2024-02-13T14:50:12.709583+0000","last_fullsized":"2024-02-13T14:50:12.709583+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:29:16.487976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710578+0000","last_change":"2024-02-13T14:49:50.030242+0000","last_active":"2024-02-13T14:50:12.710578+0000","last_peered":"2024-02-13T14:50:12.710578+0000","last_clean":"2024-02-13T14:50:12.710578+0000","last_became_active":"2024-02-13T14:49:50.029912+0000","last_became_peered":"2024-02-13T14:49:50.029912+0000","last_unstale":"2024-02-13T14:50:12.710578+0000","last_undegraded":"2024-02-13T14:50:12.710578+0000","last_fullsized":"2024-02-13T14:50:12.710578+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:45:17.200418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:19.037238+0000","last_change":"2024-02-13T14:49:48.021335+0000","last_active":"2024-02-13T14:50:19.037238+0000","last_peered":"2024-02-13T14:50:19.037238+0000","last_clean":"2024-02-13T14:50:19.037238+0000","last_became_active":"2024-02-13T14:49:48.021068+0000","last_became_peered":"2024-02-13T14:49:48.021068+0000","last_unstale":"2024-02-13T14:50:19.037238+0000","last_undegraded":"2024-02-13T14:50:19.037238+0000","last_fullsized":"2024-02-13T14:50:19.037238+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:23:18.068746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707527+0000","last_change":"2024-02-13T14:49:43.986352+0000","last_active":"2024-02-13T14:50:12.707527+0000","last_peered":"2024-02-13T14:50:12.707527+0000","last_clean":"2024-02-13T14:50:12.707527+0000","last_became_active":"2024-02-13T14:49:43.985862+0000","last_became_peered":"2024-02-13T14:49:43.985862+0000","last_unstale":"2024-02-13T14:50:12.707527+0000","last_undegraded":"2024-02-13T14:50:12.707527+0000","last_fullsized":"2024-02-13T14:50:12.707527+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:36:34.831365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709896+0000","last_change":"2024-02-13T14:49:46.002371+0000","last_active":"2024-02-13T14:50:12.709896+0000","last_peered":"2024-02-13T14:50:12.709896+0000","last_clean":"2024-02-13T14:50:12.709896+0000","last_became_active":"2024-02-13T14:49:46.002049+0000","last_became_peered":"2024-02-13T14:49:46.002049+0000","last_unstale":"2024-02-13T14:50:12.709896+0000","last_undegraded":"2024-02-13T14:50:12.709896+0000","last_fullsized":"2024-02-13T14:50:12.709896+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:37:52.548790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711464+0000","last_change":"2024-02-13T14:49:50.037137+0000","last_active":"2024-02-13T14:50:12.711464+0000","last_peered":"2024-02-13T14:50:12.711464+0000","last_clean":"2024-02-13T14:50:12.711464+0000","last_became_active":"2024-02-13T14:49:50.036413+0000","last_became_peered":"2024-02-13T14:49:50.036413+0000","last_unstale":"2024-02-13T14:50:12.711464+0000","last_undegraded":"2024-02-13T14:50:12.711464+0000","last_fullsized":"2024-02-13T14:50:12.711464+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:31:25.953981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710011+0000","last_change":"2024-02-13T14:49:48.019074+0000","last_active":"2024-02-13T14:50:12.710011+0000","last_peered":"2024-02-13T14:50:12.710011+0000","last_clean":"2024-02-13T14:50:12.710011+0000","last_became_active":"2024-02-13T14:49:48.018654+0000","last_became_peered":"2024-02-13T14:49:48.018654+0000","last_unstale":"2024-02-13T14:50:12.710011+0000","last_undegraded":"2024-02-13T14:50:12.710011+0000","last_fullsized":"2024-02-13T14:50:12.710011+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:43:11.627486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706734+0000","last_change":"2024-02-13T14:49:43.990111+0000","last_active":"2024-02-13T14:50:12.706734+0000","last_peered":"2024-02-13T14:50:12.706734+0000","last_clean":"2024-02-13T14:50:12.706734+0000","last_became_active":"2024-02-13T14:49:43.989832+0000","last_became_peered":"2024-02-13T14:49:43.989832+0000","last_unstale":"2024-02-13T14:50:12.706734+0000","last_undegraded":"2024-02-13T14:50:12.706734+0000","last_fullsized":"2024-02-13T14:50:12.706734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T14:50:27.435987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709677+0000","last_change":"2024-02-13T14:49:45.998559+0000","last_active":"2024-02-13T14:50:12.709677+0000","last_peered":"2024-02-13T14:50:12.709677+0000","last_clean":"2024-02-13T14:50:12.709677+0000","last_became_active":"2024-02-13T14:49:45.998339+0000","last_became_peered":"2024-02-13T14:49:45.998339+0000","last_unstale":"2024-02-13T14:50:12.709677+0000","last_undegraded":"2024-02-13T14:50:12.709677+0000","last_fullsized":"2024-02-13T14:50:12.709677+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T14:59:48.384896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710057+0000","last_change":"2024-02-13T14:49:50.037217+0000","last_active":"2024-02-13T14:50:12.710057+0000","last_peered":"2024-02-13T14:50:12.710057+0000","last_clean":"2024-02-13T14:50:12.710057+0000","last_became_active":"2024-02-13T14:49:50.036855+0000","last_became_peered":"2024-02-13T14:49:50.036855+0000","last_unstale":"2024-02-13T14:50:12.710057+0000","last_undegraded":"2024-02-13T14:50:12.710057+0000","last_fullsized":"2024-02-13T14:50:12.710057+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:47:14.570295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711268+0000","last_change":"2024-02-13T14:49:43.984120+0000","last_active":"2024-02-13T14:50:12.711268+0000","last_peered":"2024-02-13T14:50:12.711268+0000","last_clean":"2024-02-13T14:50:12.711268+0000","last_became_active":"2024-02-13T14:49:43.983801+0000","last_became_peered":"2024-02-13T14:49:43.983801+0000","last_unstale":"2024-02-13T14:50:12.711268+0000","last_undegraded":"2024-02-13T14:50:12.711268+0000","last_fullsized":"2024-02-13T14:50:12.711268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:07:01.039824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709239+0000","last_change":"2024-02-13T14:49:45.999416+0000","last_active":"2024-02-13T14:50:12.709239+0000","last_peered":"2024-02-13T14:50:12.709239+0000","last_clean":"2024-02-13T14:50:12.709239+0000","last_became_active":"2024-02-13T14:49:45.998986+0000","last_became_peered":"2024-02-13T14:49:45.998986+0000","last_unstale":"2024-02-13T14:50:12.709239+0000","last_undegraded":"2024-02-13T14:50:12.709239+0000","last_fullsized":"2024-02-13T14:50:12.709239+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:44:43.288856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709302+0000","last_change":"2024-02-13T14:49:48.019375+0000","last_active":"2024-02-13T14:50:12.709302+0000","last_peered":"2024-02-13T14:50:12.709302+0000","last_clean":"2024-02-13T14:50:12.709302+0000","last_became_active":"2024-02-13T14:49:48.019054+0000","last_became_peered":"2024-02-13T14:49:48.019054+0000","last_unstale":"2024-02-13T14:50:12.709302+0000","last_undegraded":"2024-02-13T14:50:12.709302+0000","last_fullsized":"2024-02-13T14:50:12.709302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:41:22.217181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707804+0000","last_change":"2024-02-13T14:49:50.034507+0000","last_active":"2024-02-13T14:50:12.707804+0000","last_peered":"2024-02-13T14:50:12.707804+0000","last_clean":"2024-02-13T14:50:12.707804+0000","last_became_active":"2024-02-13T14:49:50.033952+0000","last_became_peered":"2024-02-13T14:49:50.033952+0000","last_unstale":"2024-02-13T14:50:12.707804+0000","last_undegraded":"2024-02-13T14:50:12.707804+0000","last_fullsized":"2024-02-13T14:50:12.707804+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:07:54.700777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:19.037256+0000","last_change":"2024-02-13T14:49:48.021205+0000","last_active":"2024-02-13T14:50:19.037256+0000","last_peered":"2024-02-13T14:50:19.037256+0000","last_clean":"2024-02-13T14:50:19.037256+0000","last_became_active":"2024-02-13T14:49:48.020866+0000","last_became_peered":"2024-02-13T14:49:48.020866+0000","last_unstale":"2024-02-13T14:50:19.037256+0000","last_undegraded":"2024-02-13T14:50:19.037256+0000","last_fullsized":"2024-02-13T14:50:19.037256+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:30:53.921682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713179+0000","last_change":"2024-02-13T14:49:43.988031+0000","last_active":"2024-02-13T14:50:12.713179+0000","last_peered":"2024-02-13T14:50:12.713179+0000","last_clean":"2024-02-13T14:50:12.713179+0000","last_became_active":"2024-02-13T14:49:43.987483+0000","last_became_peered":"2024-02-13T14:49:43.987483+0000","last_unstale":"2024-02-13T14:50:12.713179+0000","last_undegraded":"2024-02-13T14:50:12.713179+0000","last_fullsized":"2024-02-13T14:50:12.713179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:37:37.979522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708698+0000","last_change":"2024-02-13T14:49:46.001560+0000","last_active":"2024-02-13T14:50:12.708698+0000","last_peered":"2024-02-13T14:50:12.708698+0000","last_clean":"2024-02-13T14:50:12.708698+0000","last_became_active":"2024-02-13T14:49:46.001003+0000","last_became_peered":"2024-02-13T14:49:46.001003+0000","last_unstale":"2024-02-13T14:50:12.708698+0000","last_undegraded":"2024-02-13T14:50:12.708698+0000","last_fullsized":"2024-02-13T14:50:12.708698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:04:16.957090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710241+0000","last_change":"2024-02-13T14:49:50.037090+0000","last_active":"2024-02-13T14:50:12.710241+0000","last_peered":"2024-02-13T14:50:12.710241+0000","last_clean":"2024-02-13T14:50:12.710241+0000","last_became_active":"2024-02-13T14:49:50.036712+0000","last_became_peered":"2024-02-13T14:49:50.036712+0000","last_unstale":"2024-02-13T14:50:12.710241+0000","last_undegraded":"2024-02-13T14:50:12.710241+0000","last_fullsized":"2024-02-13T14:50:12.710241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:54:55.701678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711656+0000","last_change":"2024-02-13T14:49:43.985406+0000","last_active":"2024-02-13T14:50:12.711656+0000","last_peered":"2024-02-13T14:50:12.711656+0000","last_clean":"2024-02-13T14:50:12.711656+0000","last_became_active":"2024-02-13T14:49:43.984553+0000","last_became_peered":"2024-02-13T14:49:43.984553+0000","last_unstale":"2024-02-13T14:50:12.711656+0000","last_undegraded":"2024-02-13T14:50:12.711656+0000","last_fullsized":"2024-02-13T14:50:12.711656+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:23:58.566162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709125+0000","last_change":"2024-02-13T14:49:46.001819+0000","last_active":"2024-02-13T14:50:12.709125+0000","last_peered":"2024-02-13T14:50:12.709125+0000","last_clean":"2024-02-13T14:50:12.709125+0000","last_became_active":"2024-02-13T14:49:46.001360+0000","last_became_peered":"2024-02-13T14:49:46.001360+0000","last_unstale":"2024-02-13T14:50:12.709125+0000","last_undegraded":"2024-02-13T14:50:12.709125+0000","last_fullsized":"2024-02-13T14:50:12.709125+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:16:26.234559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707967+0000","last_change":"2024-02-13T14:49:48.019109+0000","last_active":"2024-02-13T14:50:12.707967+0000","last_peered":"2024-02-13T14:50:12.707967+0000","last_clean":"2024-02-13T14:50:12.707967+0000","last_became_active":"2024-02-13T14:49:48.018765+0000","last_became_peered":"2024-02-13T14:49:48.018765+0000","last_unstale":"2024-02-13T14:50:12.707967+0000","last_undegraded":"2024-02-13T14:50:12.707967+0000","last_fullsized":"2024-02-13T14:50:12.707967+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:51:35.743346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706443+0000","last_change":"2024-02-13T14:49:50.028969+0000","last_active":"2024-02-13T14:50:12.706443+0000","last_peered":"2024-02-13T14:50:12.706443+0000","last_clean":"2024-02-13T14:50:12.706443+0000","last_became_active":"2024-02-13T14:49:50.028662+0000","last_became_peered":"2024-02-13T14:49:50.028662+0000","last_unstale":"2024-02-13T14:50:12.706443+0000","last_undegraded":"2024-02-13T14:50:12.706443+0000","last_fullsized":"2024-02-13T14:50:12.706443+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:24:58.629098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711676+0000","last_change":"2024-02-13T14:49:43.989780+0000","last_active":"2024-02-13T14:50:12.711676+0000","last_peered":"2024-02-13T14:50:12.711676+0000","last_clean":"2024-02-13T14:50:12.711676+0000","last_became_active":"2024-02-13T14:49:43.989245+0000","last_became_peered":"2024-02-13T14:49:43.989245+0000","last_unstale":"2024-02-13T14:50:12.711676+0000","last_undegraded":"2024-02-13T14:50:12.711676+0000","last_fullsized":"2024-02-13T14:50:12.711676+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:16:36.989280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710489+0000","last_change":"2024-02-13T14:49:46.002672+0000","last_active":"2024-02-13T14:50:12.710489+0000","last_peered":"2024-02-13T14:50:12.710489+0000","last_clean":"2024-02-13T14:50:12.710489+0000","last_became_active":"2024-02-13T14:49:46.002302+0000","last_became_peered":"2024-02-13T14:49:46.002302+0000","last_unstale":"2024-02-13T14:50:12.710489+0000","last_undegraded":"2024-02-13T14:50:12.710489+0000","last_fullsized":"2024-02-13T14:50:12.710489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:04:29.294460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707803+0000","last_change":"2024-02-13T14:49:48.017192+0000","last_active":"2024-02-13T14:50:12.707803+0000","last_peered":"2024-02-13T14:50:12.707803+0000","last_clean":"2024-02-13T14:50:12.707803+0000","last_became_active":"2024-02-13T14:49:48.016842+0000","last_became_peered":"2024-02-13T14:49:48.016842+0000","last_unstale":"2024-02-13T14:50:12.707803+0000","last_undegraded":"2024-02-13T14:50:12.707803+0000","last_fullsized":"2024-02-13T14:50:12.707803+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:14:45.811878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707506+0000","last_change":"2024-02-13T14:49:50.030701+0000","last_active":"2024-02-13T14:50:12.707506+0000","last_peered":"2024-02-13T14:50:12.707506+0000","last_clean":"2024-02-13T14:50:12.707506+0000","last_became_active":"2024-02-13T14:49:50.030248+0000","last_became_peered":"2024-02-13T14:49:50.030248+0000","last_unstale":"2024-02-13T14:50:12.707506+0000","last_undegraded":"2024-02-13T14:50:12.707506+0000","last_fullsized":"2024-02-13T14:50:12.707506+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:42:32.167292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708820+0000","last_change":"2024-02-13T14:49:43.990756+0000","last_active":"2024-02-13T14:50:12.708820+0000","last_peered":"2024-02-13T14:50:12.708820+0000","last_clean":"2024-02-13T14:50:12.708820+0000","last_became_active":"2024-02-13T14:49:43.990309+0000","last_became_peered":"2024-02-13T14:49:43.990309+0000","last_unstale":"2024-02-13T14:50:12.708820+0000","last_undegraded":"2024-02-13T14:50:12.708820+0000","last_fullsized":"2024-02-13T14:50:12.708820+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:40:42.636185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710896+0000","last_change":"2024-02-13T14:49:46.001636+0000","last_active":"2024-02-13T14:50:12.710896+0000","last_peered":"2024-02-13T14:50:12.710896+0000","last_clean":"2024-02-13T14:50:12.710896+0000","last_became_active":"2024-02-13T14:49:46.001183+0000","last_became_peered":"2024-02-13T14:49:46.001183+0000","last_unstale":"2024-02-13T14:50:12.710896+0000","last_undegraded":"2024-02-13T14:50:12.710896+0000","last_fullsized":"2024-02-13T14:50:12.710896+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:20:32.476479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710144+0000","last_change":"2024-02-13T14:49:48.017031+0000","last_active":"2024-02-13T14:50:12.710144+0000","last_peered":"2024-02-13T14:50:12.710144+0000","last_clean":"2024-02-13T14:50:12.710144+0000","last_became_active":"2024-02-13T14:49:48.016712+0000","last_became_peered":"2024-02-13T14:49:48.016712+0000","last_unstale":"2024-02-13T14:50:12.710144+0000","last_undegraded":"2024-02-13T14:50:12.710144+0000","last_fullsized":"2024-02-13T14:50:12.710144+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:54:23.016529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411343,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 14:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.34699999999999998}]},{"osd":1,"last update":"Tue Feb 13 14:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.36399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738391,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 14:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.49099999999999999}]},{"osd":2,"last update":"Tue Feb 13 14:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.40999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 14:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.38},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Tue Feb 13 14:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.51200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T14:50:34.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-13T14:50:34.761 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:35.143 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:34 smithi103 ceph-mon[25851]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-13T14:50:35.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:34 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:35.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:34 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:35.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:34 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T14:50:35.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:34 smithi103 ceph-mon[25851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T14:50:35.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:34 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:35.144 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:34 smithi103 ceph-mon[25851]: Deploying daemon grafana.a on smithi158 2024-02-13T14:50:35.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:34 smithi158 ceph-mon[27628]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-13T14:50:35.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:34 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:35.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:34 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:35.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:34 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T14:50:35.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:34 smithi158 ceph-mon[27628]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T14:50:35.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:34 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:35.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:34 smithi158 ceph-mon[27628]: Deploying daemon grafana.a on smithi158 2024-02-13T14:50:35.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:34 smithi164 ceph-mon[27636]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-13T14:50:35.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:34 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:35.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:34 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:35.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:34 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T14:50:35.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:34 smithi164 ceph-mon[27636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-13T14:50:35.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:34 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:35.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:34 smithi164 ceph-mon[27636]: Deploying daemon grafana.a on smithi158 2024-02-13T14:50:35.404 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:35 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ::ffff:172.21.15.158 - - [13/Feb/2024:14:50:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-13T14:50:35.837 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:35 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[48910]: ts=2024-02-13T14:50:35.595Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000666332s 2024-02-13T14:50:36.129 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:35 smithi103 ceph-mon[25851]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:36.180 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:35 smithi158 ceph-mon[27628]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:36.291 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:35 smithi164 ceph-mon[27636]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:50:37.185 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-02-13T14:50:37.706 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:37 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:37.706 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:37 smithi103 ceph-mon[25851]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:37.706 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:37 smithi103 ceph-mon[25851]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:50:37.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:37 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:37.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:37 smithi164 ceph-mon[27636]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:37.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:37 smithi164 ceph-mon[27636]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:50:37.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:37 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:37.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:37 smithi158 ceph-mon[27628]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:37.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:37 smithi158 ceph-mon[27628]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-13T14:50:38.864 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-02-13T14:50:36.721828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96616,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126552,"statfs":{"total":287972524032,"available":287873589248,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002774"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707944+0000","last_change":"2024-02-13T14:49:50.031906+0000","last_active":"2024-02-13T14:50:12.707944+0000","last_peered":"2024-02-13T14:50:12.707944+0000","last_clean":"2024-02-13T14:50:12.707944+0000","last_became_active":"2024-02-13T14:49:50.030827+0000","last_became_peered":"2024-02-13T14:49:50.030827+0000","last_unstale":"2024-02-13T14:50:12.707944+0000","last_undegraded":"2024-02-13T14:50:12.707944+0000","last_fullsized":"2024-02-13T14:50:12.707944+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T17:10:05.176201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709479+0000","last_change":"2024-02-13T14:49:48.016438+0000","last_active":"2024-02-13T14:50:12.709479+0000","last_peered":"2024-02-13T14:50:12.709479+0000","last_clean":"2024-02-13T14:50:12.709479+0000","last_became_active":"2024-02-13T14:49:48.015194+0000","last_became_peered":"2024-02-13T14:49:48.015194+0000","last_unstale":"2024-02-13T14:50:12.709479+0000","last_undegraded":"2024-02-13T14:50:12.709479+0000","last_fullsized":"2024-02-13T14:50:12.709479+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T17:10:29.485231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710421+0000","last_change":"2024-02-13T14:49:43.988493+0000","last_active":"2024-02-13T14:50:12.710421+0000","last_peered":"2024-02-13T14:50:12.710421+0000","last_clean":"2024-02-13T14:50:12.710421+0000","last_became_active":"2024-02-13T14:49:43.988114+0000","last_became_peered":"2024-02-13T14:49:43.988114+0000","last_unstale":"2024-02-13T14:50:12.710421+0000","last_undegraded":"2024-02-13T14:50:12.710421+0000","last_fullsized":"2024-02-13T14:50:12.710421+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:16:06.727488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709577+0000","last_change":"2024-02-13T14:49:46.001257+0000","last_active":"2024-02-13T14:50:12.709577+0000","last_peered":"2024-02-13T14:50:12.709577+0000","last_clean":"2024-02-13T14:50:12.709577+0000","last_became_active":"2024-02-13T14:49:46.000993+0000","last_became_peered":"2024-02-13T14:49:46.000993+0000","last_unstale":"2024-02-13T14:50:12.709577+0000","last_undegraded":"2024-02-13T14:50:12.709577+0000","last_fullsized":"2024-02-13T14:50:12.709577+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:33:41.628988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706714+0000","last_change":"2024-02-13T14:49:50.034415+0000","last_active":"2024-02-13T14:50:12.706714+0000","last_peered":"2024-02-13T14:50:12.706714+0000","last_clean":"2024-02-13T14:50:12.706714+0000","last_became_active":"2024-02-13T14:49:50.033995+0000","last_became_peered":"2024-02-13T14:49:50.033995+0000","last_unstale":"2024-02-13T14:50:12.706714+0000","last_undegraded":"2024-02-13T14:50:12.706714+0000","last_fullsized":"2024-02-13T14:50:12.706714+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T17:50:25.566494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707639+0000","last_change":"2024-02-13T14:49:48.020000+0000","last_active":"2024-02-13T14:50:12.707639+0000","last_peered":"2024-02-13T14:50:12.707639+0000","last_clean":"2024-02-13T14:50:12.707639+0000","last_became_active":"2024-02-13T14:49:48.019661+0000","last_became_peered":"2024-02-13T14:49:48.019661+0000","last_unstale":"2024-02-13T14:50:12.707639+0000","last_undegraded":"2024-02-13T14:50:12.707639+0000","last_fullsized":"2024-02-13T14:50:12.707639+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:55:12.042548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708969+0000","last_change":"2024-02-13T14:49:43.991078+0000","last_active":"2024-02-13T14:50:12.708969+0000","last_peered":"2024-02-13T14:50:12.708969+0000","last_clean":"2024-02-13T14:50:12.708969+0000","last_became_active":"2024-02-13T14:49:43.990692+0000","last_became_peered":"2024-02-13T14:49:43.990692+0000","last_unstale":"2024-02-13T14:50:12.708969+0000","last_undegraded":"2024-02-13T14:50:12.708969+0000","last_fullsized":"2024-02-13T14:50:12.708969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T17:55:28.171658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710091+0000","last_change":"2024-02-13T14:49:46.000738+0000","last_active":"2024-02-13T14:50:12.710091+0000","last_peered":"2024-02-13T14:50:12.710091+0000","last_clean":"2024-02-13T14:50:12.710091+0000","last_became_active":"2024-02-13T14:49:46.000360+0000","last_became_peered":"2024-02-13T14:49:46.000360+0000","last_unstale":"2024-02-13T14:50:12.710091+0000","last_undegraded":"2024-02-13T14:50:12.710091+0000","last_fullsized":"2024-02-13T14:50:12.710091+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:33:04.897697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709660+0000","last_change":"2024-02-13T14:49:50.031412+0000","last_active":"2024-02-13T14:50:12.709660+0000","last_peered":"2024-02-13T14:50:12.709660+0000","last_clean":"2024-02-13T14:50:12.709660+0000","last_became_active":"2024-02-13T14:49:50.031126+0000","last_became_peered":"2024-02-13T14:49:50.031126+0000","last_unstale":"2024-02-13T14:50:12.709660+0000","last_undegraded":"2024-02-13T14:50:12.709660+0000","last_fullsized":"2024-02-13T14:50:12.709660+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:06:41.356194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708238+0000","last_change":"2024-02-13T14:49:48.020280+0000","last_active":"2024-02-13T14:50:12.708238+0000","last_peered":"2024-02-13T14:50:12.708238+0000","last_clean":"2024-02-13T14:50:12.708238+0000","last_became_active":"2024-02-13T14:49:48.019919+0000","last_became_peered":"2024-02-13T14:49:48.019919+0000","last_unstale":"2024-02-13T14:50:12.708238+0000","last_undegraded":"2024-02-13T14:50:12.708238+0000","last_fullsized":"2024-02-13T14:50:12.708238+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:02:38.838333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713191+0000","last_change":"2024-02-13T14:49:43.984198+0000","last_active":"2024-02-13T14:50:12.713191+0000","last_peered":"2024-02-13T14:50:12.713191+0000","last_clean":"2024-02-13T14:50:12.713191+0000","last_became_active":"2024-02-13T14:49:43.983795+0000","last_became_peered":"2024-02-13T14:49:43.983795+0000","last_unstale":"2024-02-13T14:50:12.713191+0000","last_undegraded":"2024-02-13T14:50:12.713191+0000","last_fullsized":"2024-02-13T14:50:12.713191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:52:10.700900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710587+0000","last_change":"2024-02-13T14:49:46.001922+0000","last_active":"2024-02-13T14:50:12.710587+0000","last_peered":"2024-02-13T14:50:12.710587+0000","last_clean":"2024-02-13T14:50:12.710587+0000","last_became_active":"2024-02-13T14:49:46.001292+0000","last_became_peered":"2024-02-13T14:49:46.001292+0000","last_unstale":"2024-02-13T14:50:12.710587+0000","last_undegraded":"2024-02-13T14:50:12.710587+0000","last_fullsized":"2024-02-13T14:50:12.710587+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:21:25.941312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708069+0000","last_change":"2024-02-13T14:49:50.034238+0000","last_active":"2024-02-13T14:50:12.708069+0000","last_peered":"2024-02-13T14:50:12.708069+0000","last_clean":"2024-02-13T14:50:12.708069+0000","last_became_active":"2024-02-13T14:49:50.033869+0000","last_became_peered":"2024-02-13T14:49:50.033869+0000","last_unstale":"2024-02-13T14:50:12.708069+0000","last_undegraded":"2024-02-13T14:50:12.708069+0000","last_fullsized":"2024-02-13T14:50:12.708069+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:24:07.921260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708634+0000","last_change":"2024-02-13T14:49:48.019445+0000","last_active":"2024-02-13T14:50:12.708634+0000","last_peered":"2024-02-13T14:50:12.708634+0000","last_clean":"2024-02-13T14:50:12.708634+0000","last_became_active":"2024-02-13T14:49:48.019086+0000","last_became_peered":"2024-02-13T14:49:48.019086+0000","last_unstale":"2024-02-13T14:50:12.708634+0000","last_undegraded":"2024-02-13T14:50:12.708634+0000","last_fullsized":"2024-02-13T14:50:12.708634+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:22:43.864236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711095+0000","last_change":"2024-02-13T14:49:43.987991+0000","last_active":"2024-02-13T14:50:12.711095+0000","last_peered":"2024-02-13T14:50:12.711095+0000","last_clean":"2024-02-13T14:50:12.711095+0000","last_became_active":"2024-02-13T14:49:43.987676+0000","last_became_peered":"2024-02-13T14:49:43.987676+0000","last_unstale":"2024-02-13T14:50:12.711095+0000","last_undegraded":"2024-02-13T14:50:12.711095+0000","last_fullsized":"2024-02-13T14:50:12.711095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:59:41.234650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711383+0000","last_change":"2024-02-13T14:49:46.004198+0000","last_active":"2024-02-13T14:50:12.711383+0000","last_peered":"2024-02-13T14:50:12.711383+0000","last_clean":"2024-02-13T14:50:12.711383+0000","last_became_active":"2024-02-13T14:49:46.003888+0000","last_became_peered":"2024-02-13T14:49:46.003888+0000","last_unstale":"2024-02-13T14:50:12.711383+0000","last_undegraded":"2024-02-13T14:50:12.711383+0000","last_fullsized":"2024-02-13T14:50:12.711383+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:43:33.521621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709083+0000","last_change":"2024-02-13T14:49:50.031069+0000","last_active":"2024-02-13T14:50:12.709083+0000","last_peered":"2024-02-13T14:50:12.709083+0000","last_clean":"2024-02-13T14:50:12.709083+0000","last_became_active":"2024-02-13T14:49:50.030646+0000","last_became_peered":"2024-02-13T14:49:50.030646+0000","last_unstale":"2024-02-13T14:50:12.709083+0000","last_undegraded":"2024-02-13T14:50:12.709083+0000","last_fullsized":"2024-02-13T14:50:12.709083+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:00:53.592463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710215+0000","last_change":"2024-02-13T14:49:48.017156+0000","last_active":"2024-02-13T14:50:12.710215+0000","last_peered":"2024-02-13T14:50:12.710215+0000","last_clean":"2024-02-13T14:50:12.710215+0000","last_became_active":"2024-02-13T14:49:48.016677+0000","last_became_peered":"2024-02-13T14:49:48.016677+0000","last_unstale":"2024-02-13T14:50:12.710215+0000","last_undegraded":"2024-02-13T14:50:12.710215+0000","last_fullsized":"2024-02-13T14:50:12.710215+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:24:28.411282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713258+0000","last_change":"2024-02-13T14:49:43.984278+0000","last_active":"2024-02-13T14:50:12.713258+0000","last_peered":"2024-02-13T14:50:12.713258+0000","last_clean":"2024-02-13T14:50:12.713258+0000","last_became_active":"2024-02-13T14:49:43.983827+0000","last_became_peered":"2024-02-13T14:49:43.983827+0000","last_unstale":"2024-02-13T14:50:12.713258+0000","last_undegraded":"2024-02-13T14:50:12.713258+0000","last_fullsized":"2024-02-13T14:50:12.713258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:27:08.813333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711375+0000","last_change":"2024-02-13T14:49:46.004349+0000","last_active":"2024-02-13T14:50:12.711375+0000","last_peered":"2024-02-13T14:50:12.711375+0000","last_clean":"2024-02-13T14:50:12.711375+0000","last_became_active":"2024-02-13T14:49:46.004092+0000","last_became_peered":"2024-02-13T14:49:46.004092+0000","last_unstale":"2024-02-13T14:50:12.711375+0000","last_undegraded":"2024-02-13T14:50:12.711375+0000","last_fullsized":"2024-02-13T14:50:12.711375+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:00:51.594940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707856+0000","last_change":"2024-02-13T14:49:50.033339+0000","last_active":"2024-02-13T14:50:12.707856+0000","last_peered":"2024-02-13T14:50:12.707856+0000","last_clean":"2024-02-13T14:50:12.707856+0000","last_became_active":"2024-02-13T14:49:50.032842+0000","last_became_peered":"2024-02-13T14:49:50.032842+0000","last_unstale":"2024-02-13T14:50:12.707856+0000","last_undegraded":"2024-02-13T14:50:12.707856+0000","last_fullsized":"2024-02-13T14:50:12.707856+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:52:44.800408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710909+0000","last_change":"2024-02-13T14:49:48.016403+0000","last_active":"2024-02-13T14:50:12.710909+0000","last_peered":"2024-02-13T14:50:12.710909+0000","last_clean":"2024-02-13T14:50:12.710909+0000","last_became_active":"2024-02-13T14:49:48.015949+0000","last_became_peered":"2024-02-13T14:49:48.015949+0000","last_unstale":"2024-02-13T14:50:12.710909+0000","last_undegraded":"2024-02-13T14:50:12.710909+0000","last_fullsized":"2024-02-13T14:50:12.710909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:15:17.531499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708248+0000","last_change":"2024-02-13T14:49:43.984659+0000","last_active":"2024-02-13T14:50:12.708248+0000","last_peered":"2024-02-13T14:50:12.708248+0000","last_clean":"2024-02-13T14:50:12.708248+0000","last_became_active":"2024-02-13T14:49:43.983925+0000","last_became_peered":"2024-02-13T14:49:43.983925+0000","last_unstale":"2024-02-13T14:50:12.708248+0000","last_undegraded":"2024-02-13T14:50:12.708248+0000","last_fullsized":"2024-02-13T14:50:12.708248+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:01:09.481219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713082+0000","last_change":"2024-02-13T14:49:46.001887+0000","last_active":"2024-02-13T14:50:12.713082+0000","last_peered":"2024-02-13T14:50:12.713082+0000","last_clean":"2024-02-13T14:50:12.713082+0000","last_became_active":"2024-02-13T14:49:46.001598+0000","last_became_peered":"2024-02-13T14:49:46.001598+0000","last_unstale":"2024-02-13T14:50:12.713082+0000","last_undegraded":"2024-02-13T14:50:12.713082+0000","last_fullsized":"2024-02-13T14:50:12.713082+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:44:34.051990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707625+0000","last_change":"2024-02-13T14:49:50.033762+0000","last_active":"2024-02-13T14:50:12.707625+0000","last_peered":"2024-02-13T14:50:12.707625+0000","last_clean":"2024-02-13T14:50:12.707625+0000","last_became_active":"2024-02-13T14:49:50.033372+0000","last_became_peered":"2024-02-13T14:49:50.033372+0000","last_unstale":"2024-02-13T14:50:12.707625+0000","last_undegraded":"2024-02-13T14:50:12.707625+0000","last_fullsized":"2024-02-13T14:50:12.707625+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:50:34.269216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:24.038112+0000","last_change":"2024-02-13T14:49:48.017951+0000","last_active":"2024-02-13T14:50:24.038112+0000","last_peered":"2024-02-13T14:50:24.038112+0000","last_clean":"2024-02-13T14:50:24.038112+0000","last_became_active":"2024-02-13T14:49:48.016843+0000","last_became_peered":"2024-02-13T14:49:48.016843+0000","last_unstale":"2024-02-13T14:50:24.038112+0000","last_undegraded":"2024-02-13T14:50:24.038112+0000","last_fullsized":"2024-02-13T14:50:24.038112+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:47:27.407906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706833+0000","last_change":"2024-02-13T14:49:43.987751+0000","last_active":"2024-02-13T14:50:12.706833+0000","last_peered":"2024-02-13T14:50:12.706833+0000","last_clean":"2024-02-13T14:50:12.706833+0000","last_became_active":"2024-02-13T14:49:43.987160+0000","last_became_peered":"2024-02-13T14:49:43.987160+0000","last_unstale":"2024-02-13T14:50:12.706833+0000","last_undegraded":"2024-02-13T14:50:12.706833+0000","last_fullsized":"2024-02-13T14:50:12.706833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:50:19.697796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706557+0000","last_change":"2024-02-13T14:49:46.001845+0000","last_active":"2024-02-13T14:50:12.706557+0000","last_peered":"2024-02-13T14:50:12.706557+0000","last_clean":"2024-02-13T14:50:12.706557+0000","last_became_active":"2024-02-13T14:49:46.000969+0000","last_became_peered":"2024-02-13T14:49:46.000969+0000","last_unstale":"2024-02-13T14:50:12.706557+0000","last_undegraded":"2024-02-13T14:50:12.706557+0000","last_fullsized":"2024-02-13T14:50:12.706557+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:31:39.856295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710449+0000","last_change":"2024-02-13T14:49:50.032319+0000","last_active":"2024-02-13T14:50:12.710449+0000","last_peered":"2024-02-13T14:50:12.710449+0000","last_clean":"2024-02-13T14:50:12.710449+0000","last_became_active":"2024-02-13T14:49:50.031860+0000","last_became_peered":"2024-02-13T14:49:50.031860+0000","last_unstale":"2024-02-13T14:50:12.710449+0000","last_undegraded":"2024-02-13T14:50:12.710449+0000","last_fullsized":"2024-02-13T14:50:12.710449+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:43:02.044782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707924+0000","last_change":"2024-02-13T14:49:48.018584+0000","last_active":"2024-02-13T14:50:12.707924+0000","last_peered":"2024-02-13T14:50:12.707924+0000","last_clean":"2024-02-13T14:50:12.707924+0000","last_became_active":"2024-02-13T14:49:48.017483+0000","last_became_peered":"2024-02-13T14:49:48.017483+0000","last_unstale":"2024-02-13T14:50:12.707924+0000","last_undegraded":"2024-02-13T14:50:12.707924+0000","last_fullsized":"2024-02-13T14:50:12.707924+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:01:16.753767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709425+0000","last_change":"2024-02-13T14:49:43.984699+0000","last_active":"2024-02-13T14:50:12.709425+0000","last_peered":"2024-02-13T14:50:12.709425+0000","last_clean":"2024-02-13T14:50:12.709425+0000","last_became_active":"2024-02-13T14:49:43.984415+0000","last_became_peered":"2024-02-13T14:49:43.984415+0000","last_unstale":"2024-02-13T14:50:12.709425+0000","last_undegraded":"2024-02-13T14:50:12.709425+0000","last_fullsized":"2024-02-13T14:50:12.709425+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:17:05.997498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706583+0000","last_change":"2024-02-13T14:49:45.999543+0000","last_active":"2024-02-13T14:50:12.706583+0000","last_peered":"2024-02-13T14:50:12.706583+0000","last_clean":"2024-02-13T14:50:12.706583+0000","last_became_active":"2024-02-13T14:49:45.998762+0000","last_became_peered":"2024-02-13T14:49:45.998762+0000","last_unstale":"2024-02-13T14:50:12.706583+0000","last_undegraded":"2024-02-13T14:50:12.706583+0000","last_fullsized":"2024-02-13T14:50:12.706583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:19:13.130882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707824+0000","last_change":"2024-02-13T14:49:50.035304+0000","last_active":"2024-02-13T14:50:12.707824+0000","last_peered":"2024-02-13T14:50:12.707824+0000","last_clean":"2024-02-13T14:50:12.707824+0000","last_became_active":"2024-02-13T14:49:50.034994+0000","last_became_peered":"2024-02-13T14:49:50.034994+0000","last_unstale":"2024-02-13T14:50:12.707824+0000","last_undegraded":"2024-02-13T14:50:12.707824+0000","last_fullsized":"2024-02-13T14:50:12.707824+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:32:26.322389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706485+0000","last_change":"2024-02-13T14:49:48.016645+0000","last_active":"2024-02-13T14:50:12.706485+0000","last_peered":"2024-02-13T14:50:12.706485+0000","last_clean":"2024-02-13T14:50:12.706485+0000","last_became_active":"2024-02-13T14:49:48.016091+0000","last_became_peered":"2024-02-13T14:49:48.016091+0000","last_unstale":"2024-02-13T14:50:12.706485+0000","last_undegraded":"2024-02-13T14:50:12.706485+0000","last_fullsized":"2024-02-13T14:50:12.706485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:56:06.773843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711001+0000","last_change":"2024-02-13T14:49:43.986023+0000","last_active":"2024-02-13T14:50:12.711001+0000","last_peered":"2024-02-13T14:50:12.711001+0000","last_clean":"2024-02-13T14:50:12.711001+0000","last_became_active":"2024-02-13T14:49:43.985584+0000","last_became_peered":"2024-02-13T14:49:43.985584+0000","last_unstale":"2024-02-13T14:50:12.711001+0000","last_undegraded":"2024-02-13T14:50:12.711001+0000","last_fullsized":"2024-02-13T14:50:12.711001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:57:34.370906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710463+0000","last_change":"2024-02-13T14:49:46.003146+0000","last_active":"2024-02-13T14:50:12.710463+0000","last_peered":"2024-02-13T14:50:12.710463+0000","last_clean":"2024-02-13T14:50:12.710463+0000","last_became_active":"2024-02-13T14:49:46.002683+0000","last_became_peered":"2024-02-13T14:49:46.002683+0000","last_unstale":"2024-02-13T14:50:12.710463+0000","last_undegraded":"2024-02-13T14:50:12.710463+0000","last_fullsized":"2024-02-13T14:50:12.710463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:09:12.685827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707896+0000","last_change":"2024-02-13T14:49:50.031110+0000","last_active":"2024-02-13T14:50:12.707896+0000","last_peered":"2024-02-13T14:50:12.707896+0000","last_clean":"2024-02-13T14:50:12.707896+0000","last_became_active":"2024-02-13T14:49:50.030633+0000","last_became_peered":"2024-02-13T14:49:50.030633+0000","last_unstale":"2024-02-13T14:50:12.707896+0000","last_undegraded":"2024-02-13T14:50:12.707896+0000","last_fullsized":"2024-02-13T14:50:12.707896+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T14:50:44.378879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707006+0000","last_change":"2024-02-13T14:49:48.016354+0000","last_active":"2024-02-13T14:50:12.707006+0000","last_peered":"2024-02-13T14:50:12.707006+0000","last_clean":"2024-02-13T14:50:12.707006+0000","last_became_active":"2024-02-13T14:49:48.015977+0000","last_became_peered":"2024-02-13T14:49:48.015977+0000","last_unstale":"2024-02-13T14:50:12.707006+0000","last_undegraded":"2024-02-13T14:50:12.707006+0000","last_fullsized":"2024-02-13T14:50:12.707006+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:01:52.382248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709193+0000","last_change":"2024-02-13T14:49:43.984234+0000","last_active":"2024-02-13T14:50:12.709193+0000","last_peered":"2024-02-13T14:50:12.709193+0000","last_clean":"2024-02-13T14:50:12.709193+0000","last_became_active":"2024-02-13T14:49:43.983665+0000","last_became_peered":"2024-02-13T14:49:43.983665+0000","last_unstale":"2024-02-13T14:50:12.709193+0000","last_undegraded":"2024-02-13T14:50:12.709193+0000","last_fullsized":"2024-02-13T14:50:12.709193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:16:47.899383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710288+0000","last_change":"2024-02-13T14:49:46.003062+0000","last_active":"2024-02-13T14:50:12.710288+0000","last_peered":"2024-02-13T14:50:12.710288+0000","last_clean":"2024-02-13T14:50:12.710288+0000","last_became_active":"2024-02-13T14:49:46.002748+0000","last_became_peered":"2024-02-13T14:49:46.002748+0000","last_unstale":"2024-02-13T14:50:12.710288+0000","last_undegraded":"2024-02-13T14:50:12.710288+0000","last_fullsized":"2024-02-13T14:50:12.710288+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:57:33.201273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709909+0000","last_change":"2024-02-13T14:49:50.036030+0000","last_active":"2024-02-13T14:50:12.709909+0000","last_peered":"2024-02-13T14:50:12.709909+0000","last_clean":"2024-02-13T14:50:12.709909+0000","last_became_active":"2024-02-13T14:49:50.035721+0000","last_became_peered":"2024-02-13T14:49:50.035721+0000","last_unstale":"2024-02-13T14:50:12.709909+0000","last_undegraded":"2024-02-13T14:50:12.709909+0000","last_fullsized":"2024-02-13T14:50:12.709909+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:30:24.686800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708334+0000","last_change":"2024-02-13T14:49:48.019415+0000","last_active":"2024-02-13T14:50:12.708334+0000","last_peered":"2024-02-13T14:50:12.708334+0000","last_clean":"2024-02-13T14:50:12.708334+0000","last_became_active":"2024-02-13T14:49:48.019051+0000","last_became_peered":"2024-02-13T14:49:48.019051+0000","last_unstale":"2024-02-13T14:50:12.708334+0000","last_undegraded":"2024-02-13T14:50:12.708334+0000","last_fullsized":"2024-02-13T14:50:12.708334+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:52:02.151270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713276+0000","last_change":"2024-02-13T14:49:43.989768+0000","last_active":"2024-02-13T14:50:12.713276+0000","last_peered":"2024-02-13T14:50:12.713276+0000","last_clean":"2024-02-13T14:50:12.713276+0000","last_became_active":"2024-02-13T14:49:43.988724+0000","last_became_peered":"2024-02-13T14:49:43.988724+0000","last_unstale":"2024-02-13T14:50:12.713276+0000","last_undegraded":"2024-02-13T14:50:12.713276+0000","last_fullsized":"2024-02-13T14:50:12.713276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:37:16.142844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710755+0000","last_change":"2024-02-13T14:49:46.001807+0000","last_active":"2024-02-13T14:50:12.710755+0000","last_peered":"2024-02-13T14:50:12.710755+0000","last_clean":"2024-02-13T14:50:12.710755+0000","last_became_active":"2024-02-13T14:49:46.001317+0000","last_became_peered":"2024-02-13T14:49:46.001317+0000","last_unstale":"2024-02-13T14:50:12.710755+0000","last_undegraded":"2024-02-13T14:50:12.710755+0000","last_fullsized":"2024-02-13T14:50:12.710755+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:09:05.961764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709690+0000","last_change":"2024-02-13T14:49:50.035672+0000","last_active":"2024-02-13T14:50:12.709690+0000","last_peered":"2024-02-13T14:50:12.709690+0000","last_clean":"2024-02-13T14:50:12.709690+0000","last_became_active":"2024-02-13T14:49:50.035371+0000","last_became_peered":"2024-02-13T14:49:50.035371+0000","last_unstale":"2024-02-13T14:50:12.709690+0000","last_undegraded":"2024-02-13T14:50:12.709690+0000","last_fullsized":"2024-02-13T14:50:12.709690+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:22:15.419067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706775+0000","last_change":"2024-02-13T14:49:48.016954+0000","last_active":"2024-02-13T14:50:12.706775+0000","last_peered":"2024-02-13T14:50:12.706775+0000","last_clean":"2024-02-13T14:50:12.706775+0000","last_became_active":"2024-02-13T14:49:48.016611+0000","last_became_peered":"2024-02-13T14:49:48.016611+0000","last_unstale":"2024-02-13T14:50:12.706775+0000","last_undegraded":"2024-02-13T14:50:12.706775+0000","last_fullsized":"2024-02-13T14:50:12.706775+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:19:38.862691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710466+0000","last_change":"2024-02-13T14:49:43.985982+0000","last_active":"2024-02-13T14:50:12.710466+0000","last_peered":"2024-02-13T14:50:12.710466+0000","last_clean":"2024-02-13T14:50:12.710466+0000","last_became_active":"2024-02-13T14:49:43.985643+0000","last_became_peered":"2024-02-13T14:49:43.985643+0000","last_unstale":"2024-02-13T14:50:12.710466+0000","last_undegraded":"2024-02-13T14:50:12.710466+0000","last_fullsized":"2024-02-13T14:50:12.710466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:46:56.065502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711549+0000","last_change":"2024-02-13T14:49:46.002295+0000","last_active":"2024-02-13T14:50:12.711549+0000","last_peered":"2024-02-13T14:50:12.711549+0000","last_clean":"2024-02-13T14:50:12.711549+0000","last_became_active":"2024-02-13T14:49:46.001624+0000","last_became_peered":"2024-02-13T14:49:46.001624+0000","last_unstale":"2024-02-13T14:50:12.711549+0000","last_undegraded":"2024-02-13T14:50:12.711549+0000","last_fullsized":"2024-02-13T14:50:12.711549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:04:31.089130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708035+0000","last_change":"2024-02-13T14:49:50.034707+0000","last_active":"2024-02-13T14:50:12.708035+0000","last_peered":"2024-02-13T14:50:12.708035+0000","last_clean":"2024-02-13T14:50:12.708035+0000","last_became_active":"2024-02-13T14:49:50.034052+0000","last_became_peered":"2024-02-13T14:49:50.034052+0000","last_unstale":"2024-02-13T14:50:12.708035+0000","last_undegraded":"2024-02-13T14:50:12.708035+0000","last_fullsized":"2024-02-13T14:50:12.708035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:45:49.401829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:24.038138+0000","last_change":"2024-02-13T14:49:48.015793+0000","last_active":"2024-02-13T14:50:24.038138+0000","last_peered":"2024-02-13T14:50:24.038138+0000","last_clean":"2024-02-13T14:50:24.038138+0000","last_became_active":"2024-02-13T14:49:48.015494+0000","last_became_peered":"2024-02-13T14:49:48.015494+0000","last_unstale":"2024-02-13T14:50:24.038138+0000","last_undegraded":"2024-02-13T14:50:24.038138+0000","last_fullsized":"2024-02-13T14:50:24.038138+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:38:18.327521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713420+0000","last_change":"2024-02-13T14:49:43.988992+0000","last_active":"2024-02-13T14:50:12.713420+0000","last_peered":"2024-02-13T14:50:12.713420+0000","last_clean":"2024-02-13T14:50:12.713420+0000","last_became_active":"2024-02-13T14:49:43.988704+0000","last_became_peered":"2024-02-13T14:49:43.988704+0000","last_unstale":"2024-02-13T14:50:12.713420+0000","last_undegraded":"2024-02-13T14:50:12.713420+0000","last_fullsized":"2024-02-13T14:50:12.713420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:17:17.114027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707231+0000","last_change":"2024-02-13T14:49:46.001820+0000","last_active":"2024-02-13T14:50:12.707231+0000","last_peered":"2024-02-13T14:50:12.707231+0000","last_clean":"2024-02-13T14:50:12.707231+0000","last_became_active":"2024-02-13T14:49:46.001267+0000","last_became_peered":"2024-02-13T14:49:46.001267+0000","last_unstale":"2024-02-13T14:50:12.707231+0000","last_undegraded":"2024-02-13T14:50:12.707231+0000","last_fullsized":"2024-02-13T14:50:12.707231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:31:12.315257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708259+0000","last_change":"2024-02-13T14:49:50.034532+0000","last_active":"2024-02-13T14:50:12.708259+0000","last_peered":"2024-02-13T14:50:12.708259+0000","last_clean":"2024-02-13T14:50:12.708259+0000","last_became_active":"2024-02-13T14:49:50.034158+0000","last_became_peered":"2024-02-13T14:49:50.034158+0000","last_unstale":"2024-02-13T14:50:12.708259+0000","last_undegraded":"2024-02-13T14:50:12.708259+0000","last_fullsized":"2024-02-13T14:50:12.708259+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:27:01.275495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:24.038174+0000","last_change":"2024-02-13T14:49:48.016146+0000","last_active":"2024-02-13T14:50:24.038174+0000","last_peered":"2024-02-13T14:50:24.038174+0000","last_clean":"2024-02-13T14:50:24.038174+0000","last_became_active":"2024-02-13T14:49:48.015330+0000","last_became_peered":"2024-02-13T14:49:48.015330+0000","last_unstale":"2024-02-13T14:50:24.038174+0000","last_undegraded":"2024-02-13T14:50:24.038174+0000","last_fullsized":"2024-02-13T14:50:24.038174+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:27:54.322942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710908+0000","last_change":"2024-02-13T14:49:43.987875+0000","last_active":"2024-02-13T14:50:12.710908+0000","last_peered":"2024-02-13T14:50:12.710908+0000","last_clean":"2024-02-13T14:50:12.710908+0000","last_became_active":"2024-02-13T14:49:43.987560+0000","last_became_peered":"2024-02-13T14:49:43.987560+0000","last_unstale":"2024-02-13T14:50:12.710908+0000","last_undegraded":"2024-02-13T14:50:12.710908+0000","last_fullsized":"2024-02-13T14:50:12.710908+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:35:56.749045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708817+0000","last_change":"2024-02-13T14:49:46.001736+0000","last_active":"2024-02-13T14:50:12.708817+0000","last_peered":"2024-02-13T14:50:12.708817+0000","last_clean":"2024-02-13T14:50:12.708817+0000","last_became_active":"2024-02-13T14:49:46.001264+0000","last_became_peered":"2024-02-13T14:49:46.001264+0000","last_unstale":"2024-02-13T14:50:12.708817+0000","last_undegraded":"2024-02-13T14:50:12.708817+0000","last_fullsized":"2024-02-13T14:50:12.708817+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:30:11.936954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708724+0000","last_change":"2024-02-13T14:49:50.031258+0000","last_active":"2024-02-13T14:50:12.708724+0000","last_peered":"2024-02-13T14:50:12.708724+0000","last_clean":"2024-02-13T14:50:12.708724+0000","last_became_active":"2024-02-13T14:49:50.030786+0000","last_became_peered":"2024-02-13T14:49:50.030786+0000","last_unstale":"2024-02-13T14:50:12.708724+0000","last_undegraded":"2024-02-13T14:50:12.708724+0000","last_fullsized":"2024-02-13T14:50:12.708724+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:59:04.844751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709929+0000","last_change":"2024-02-13T14:49:48.019153+0000","last_active":"2024-02-13T14:50:12.709929+0000","last_peered":"2024-02-13T14:50:12.709929+0000","last_clean":"2024-02-13T14:50:12.709929+0000","last_became_active":"2024-02-13T14:49:48.018803+0000","last_became_peered":"2024-02-13T14:49:48.018803+0000","last_unstale":"2024-02-13T14:50:12.709929+0000","last_undegraded":"2024-02-13T14:50:12.709929+0000","last_fullsized":"2024-02-13T14:50:12.709929+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:43:54.058564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707758+0000","last_change":"2024-02-13T14:49:43.983981+0000","last_active":"2024-02-13T14:50:12.707758+0000","last_peered":"2024-02-13T14:50:12.707758+0000","last_clean":"2024-02-13T14:50:12.707758+0000","last_became_active":"2024-02-13T14:49:43.983571+0000","last_became_peered":"2024-02-13T14:49:43.983571+0000","last_unstale":"2024-02-13T14:50:12.707758+0000","last_undegraded":"2024-02-13T14:50:12.707758+0000","last_fullsized":"2024-02-13T14:50:12.707758+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:51:35.609959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707336+0000","last_change":"2024-02-13T14:49:46.002760+0000","last_active":"2024-02-13T14:50:12.707336+0000","last_peered":"2024-02-13T14:50:12.707336+0000","last_clean":"2024-02-13T14:50:12.707336+0000","last_became_active":"2024-02-13T14:49:46.002205+0000","last_became_peered":"2024-02-13T14:49:46.002205+0000","last_unstale":"2024-02-13T14:50:12.707336+0000","last_undegraded":"2024-02-13T14:50:12.707336+0000","last_fullsized":"2024-02-13T14:50:12.707336+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:30:02.807948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709686+0000","last_change":"2024-02-13T14:49:50.032823+0000","last_active":"2024-02-13T14:50:12.709686+0000","last_peered":"2024-02-13T14:50:12.709686+0000","last_clean":"2024-02-13T14:50:12.709686+0000","last_became_active":"2024-02-13T14:49:50.032454+0000","last_became_peered":"2024-02-13T14:49:50.032454+0000","last_unstale":"2024-02-13T14:50:12.709686+0000","last_undegraded":"2024-02-13T14:50:12.709686+0000","last_fullsized":"2024-02-13T14:50:12.709686+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:26:18.706216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708016+0000","last_change":"2024-02-13T14:49:48.018734+0000","last_active":"2024-02-13T14:50:12.708016+0000","last_peered":"2024-02-13T14:50:12.708016+0000","last_clean":"2024-02-13T14:50:12.708016+0000","last_became_active":"2024-02-13T14:49:48.018342+0000","last_became_peered":"2024-02-13T14:49:48.018342+0000","last_unstale":"2024-02-13T14:50:12.708016+0000","last_undegraded":"2024-02-13T14:50:12.708016+0000","last_fullsized":"2024-02-13T14:50:12.708016+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:33:41.799571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709533+0000","last_change":"2024-02-13T14:49:43.987748+0000","last_active":"2024-02-13T14:50:12.709533+0000","last_peered":"2024-02-13T14:50:12.709533+0000","last_clean":"2024-02-13T14:50:12.709533+0000","last_became_active":"2024-02-13T14:49:43.987463+0000","last_became_peered":"2024-02-13T14:49:43.987463+0000","last_unstale":"2024-02-13T14:50:12.709533+0000","last_undegraded":"2024-02-13T14:50:12.709533+0000","last_fullsized":"2024-02-13T14:50:12.709533+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:14:45.012311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710328+0000","last_change":"2024-02-13T14:49:46.002448+0000","last_active":"2024-02-13T14:50:12.710328+0000","last_peered":"2024-02-13T14:50:12.710328+0000","last_clean":"2024-02-13T14:50:12.710328+0000","last_became_active":"2024-02-13T14:49:46.002205+0000","last_became_peered":"2024-02-13T14:49:46.002205+0000","last_unstale":"2024-02-13T14:50:12.710328+0000","last_undegraded":"2024-02-13T14:50:12.710328+0000","last_fullsized":"2024-02-13T14:50:12.710328+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:17:11.020957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711181+0000","last_change":"2024-02-13T14:49:50.037297+0000","last_active":"2024-02-13T14:50:12.711181+0000","last_peered":"2024-02-13T14:50:12.711181+0000","last_clean":"2024-02-13T14:50:12.711181+0000","last_became_active":"2024-02-13T14:49:50.036878+0000","last_became_peered":"2024-02-13T14:49:50.036878+0000","last_unstale":"2024-02-13T14:50:12.711181+0000","last_undegraded":"2024-02-13T14:50:12.711181+0000","last_fullsized":"2024-02-13T14:50:12.711181+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:35:36.643689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713599+0000","last_change":"2024-02-13T14:49:43.984783+0000","last_active":"2024-02-13T14:50:12.713599+0000","last_peered":"2024-02-13T14:50:12.713599+0000","last_clean":"2024-02-13T14:50:12.713599+0000","last_became_active":"2024-02-13T14:49:43.984004+0000","last_became_peered":"2024-02-13T14:49:43.984004+0000","last_unstale":"2024-02-13T14:50:12.713599+0000","last_undegraded":"2024-02-13T14:50:12.713599+0000","last_fullsized":"2024-02-13T14:50:12.713599+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:57:33.524731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711204+0000","last_change":"2024-02-13T14:49:46.002699+0000","last_active":"2024-02-13T14:50:12.711204+0000","last_peered":"2024-02-13T14:50:12.711204+0000","last_clean":"2024-02-13T14:50:12.711204+0000","last_became_active":"2024-02-13T14:49:46.002401+0000","last_became_peered":"2024-02-13T14:49:46.002401+0000","last_unstale":"2024-02-13T14:50:12.711204+0000","last_undegraded":"2024-02-13T14:50:12.711204+0000","last_fullsized":"2024-02-13T14:50:12.711204+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:39:29.043303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:24.038210+0000","last_change":"2024-02-13T14:49:48.015846+0000","last_active":"2024-02-13T14:50:24.038210+0000","last_peered":"2024-02-13T14:50:24.038210+0000","last_clean":"2024-02-13T14:50:24.038210+0000","last_became_active":"2024-02-13T14:49:48.015286+0000","last_became_peered":"2024-02-13T14:49:48.015286+0000","last_unstale":"2024-02-13T14:50:24.038210+0000","last_undegraded":"2024-02-13T14:50:24.038210+0000","last_fullsized":"2024-02-13T14:50:24.038210+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:23:29.086268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707523+0000","last_change":"2024-02-13T14:49:50.028918+0000","last_active":"2024-02-13T14:50:12.707523+0000","last_peered":"2024-02-13T14:50:12.707523+0000","last_clean":"2024-02-13T14:50:12.707523+0000","last_became_active":"2024-02-13T14:49:50.028199+0000","last_became_peered":"2024-02-13T14:49:50.028199+0000","last_unstale":"2024-02-13T14:50:12.707523+0000","last_undegraded":"2024-02-13T14:50:12.707523+0000","last_fullsized":"2024-02-13T14:50:12.707523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:01:00.591300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709346+0000","last_change":"2024-02-13T14:49:43.988261+0000","last_active":"2024-02-13T14:50:12.709346+0000","last_peered":"2024-02-13T14:50:12.709346+0000","last_clean":"2024-02-13T14:50:12.709346+0000","last_became_active":"2024-02-13T14:49:43.987854+0000","last_became_peered":"2024-02-13T14:49:43.987854+0000","last_unstale":"2024-02-13T14:50:12.709346+0000","last_undegraded":"2024-02-13T14:50:12.709346+0000","last_fullsized":"2024-02-13T14:50:12.709346+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:48:44.471710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:24.038318+0000","last_change":"2024-02-13T14:49:46.002251+0000","last_active":"2024-02-13T14:50:24.038318+0000","last_peered":"2024-02-13T14:50:24.038318+0000","last_clean":"2024-02-13T14:50:24.038318+0000","last_became_active":"2024-02-13T14:49:46.001489+0000","last_became_peered":"2024-02-13T14:49:46.001489+0000","last_unstale":"2024-02-13T14:50:24.038318+0000","last_undegraded":"2024-02-13T14:50:24.038318+0000","last_fullsized":"2024-02-13T14:50:24.038318+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:30:11.548080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710154+0000","last_change":"2024-02-13T14:49:48.016152+0000","last_active":"2024-02-13T14:50:12.710154+0000","last_peered":"2024-02-13T14:50:12.710154+0000","last_clean":"2024-02-13T14:50:12.710154+0000","last_became_active":"2024-02-13T14:49:48.015597+0000","last_became_peered":"2024-02-13T14:49:48.015597+0000","last_unstale":"2024-02-13T14:50:12.710154+0000","last_undegraded":"2024-02-13T14:50:12.710154+0000","last_fullsized":"2024-02-13T14:50:12.710154+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:32:41.154762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708504+0000","last_change":"2024-02-13T14:49:50.033013+0000","last_active":"2024-02-13T14:50:12.708504+0000","last_peered":"2024-02-13T14:50:12.708504+0000","last_clean":"2024-02-13T14:50:12.708504+0000","last_became_active":"2024-02-13T14:49:50.032502+0000","last_became_peered":"2024-02-13T14:49:50.032502+0000","last_unstale":"2024-02-13T14:50:12.708504+0000","last_undegraded":"2024-02-13T14:50:12.708504+0000","last_fullsized":"2024-02-13T14:50:12.708504+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:44:59.486399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708428+0000","last_change":"2024-02-13T14:49:43.984911+0000","last_active":"2024-02-13T14:50:12.708428+0000","last_peered":"2024-02-13T14:50:12.708428+0000","last_clean":"2024-02-13T14:50:12.708428+0000","last_became_active":"2024-02-13T14:49:43.984530+0000","last_became_peered":"2024-02-13T14:49:43.984530+0000","last_unstale":"2024-02-13T14:50:12.708428+0000","last_undegraded":"2024-02-13T14:50:12.708428+0000","last_fullsized":"2024-02-13T14:50:12.708428+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:06:22.508793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708900+0000","last_change":"2024-02-13T14:49:46.002011+0000","last_active":"2024-02-13T14:50:12.708900+0000","last_peered":"2024-02-13T14:50:12.708900+0000","last_clean":"2024-02-13T14:50:12.708900+0000","last_became_active":"2024-02-13T14:49:46.001146+0000","last_became_peered":"2024-02-13T14:49:46.001146+0000","last_unstale":"2024-02-13T14:50:12.708900+0000","last_undegraded":"2024-02-13T14:50:12.708900+0000","last_fullsized":"2024-02-13T14:50:12.708900+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:58:44.212130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708802+0000","last_change":"2024-02-13T14:49:48.017518+0000","last_active":"2024-02-13T14:50:12.708802+0000","last_peered":"2024-02-13T14:50:12.708802+0000","last_clean":"2024-02-13T14:50:12.708802+0000","last_became_active":"2024-02-13T14:49:48.017207+0000","last_became_peered":"2024-02-13T14:49:48.017207+0000","last_unstale":"2024-02-13T14:50:12.708802+0000","last_undegraded":"2024-02-13T14:50:12.708802+0000","last_fullsized":"2024-02-13T14:50:12.708802+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:04:13.651245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708674+0000","last_change":"2024-02-13T14:49:50.034965+0000","last_active":"2024-02-13T14:50:12.708674+0000","last_peered":"2024-02-13T14:50:12.708674+0000","last_clean":"2024-02-13T14:50:12.708674+0000","last_became_active":"2024-02-13T14:49:50.034624+0000","last_became_peered":"2024-02-13T14:49:50.034624+0000","last_unstale":"2024-02-13T14:50:12.708674+0000","last_undegraded":"2024-02-13T14:50:12.708674+0000","last_fullsized":"2024-02-13T14:50:12.708674+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:06:08.558445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709558+0000","last_change":"2024-02-13T14:49:46.002350+0000","last_active":"2024-02-13T14:50:12.709558+0000","last_peered":"2024-02-13T14:50:12.709558+0000","last_clean":"2024-02-13T14:50:12.709558+0000","last_became_active":"2024-02-13T14:49:46.002053+0000","last_became_peered":"2024-02-13T14:49:46.002053+0000","last_unstale":"2024-02-13T14:50:12.709558+0000","last_undegraded":"2024-02-13T14:50:12.709558+0000","last_fullsized":"2024-02-13T14:50:12.709558+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:30:11.897592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706645+0000","last_change":"2024-02-13T14:49:43.987540+0000","last_active":"2024-02-13T14:50:12.706645+0000","last_peered":"2024-02-13T14:50:12.706645+0000","last_clean":"2024-02-13T14:50:12.706645+0000","last_became_active":"2024-02-13T14:49:43.987052+0000","last_became_peered":"2024-02-13T14:49:43.987052+0000","last_unstale":"2024-02-13T14:50:12.706645+0000","last_undegraded":"2024-02-13T14:50:12.706645+0000","last_fullsized":"2024-02-13T14:50:12.706645+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:38:56.196725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710206+0000","last_change":"2024-02-13T14:49:48.020513+0000","last_active":"2024-02-13T14:50:12.710206+0000","last_peered":"2024-02-13T14:50:12.710206+0000","last_clean":"2024-02-13T14:50:12.710206+0000","last_became_active":"2024-02-13T14:49:48.019602+0000","last_became_peered":"2024-02-13T14:49:48.019602+0000","last_unstale":"2024-02-13T14:50:12.710206+0000","last_undegraded":"2024-02-13T14:50:12.710206+0000","last_fullsized":"2024-02-13T14:50:12.710206+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:26:08.069433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709861+0000","last_change":"2024-02-13T14:49:50.037294+0000","last_active":"2024-02-13T14:50:12.709861+0000","last_peered":"2024-02-13T14:50:12.709861+0000","last_clean":"2024-02-13T14:50:12.709861+0000","last_became_active":"2024-02-13T14:49:50.036614+0000","last_became_peered":"2024-02-13T14:49:50.036614+0000","last_unstale":"2024-02-13T14:50:12.709861+0000","last_undegraded":"2024-02-13T14:50:12.709861+0000","last_fullsized":"2024-02-13T14:50:12.709861+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T17:57:00.459745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707050+0000","last_change":"2024-02-13T14:49:46.001762+0000","last_active":"2024-02-13T14:50:12.707050+0000","last_peered":"2024-02-13T14:50:12.707050+0000","last_clean":"2024-02-13T14:50:12.707050+0000","last_became_active":"2024-02-13T14:49:46.001221+0000","last_became_peered":"2024-02-13T14:49:46.001221+0000","last_unstale":"2024-02-13T14:50:12.707050+0000","last_undegraded":"2024-02-13T14:50:12.707050+0000","last_fullsized":"2024-02-13T14:50:12.707050+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:31:58.993425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708651+0000","last_change":"2024-02-13T14:49:43.982793+0000","last_active":"2024-02-13T14:50:12.708651+0000","last_peered":"2024-02-13T14:50:12.708651+0000","last_clean":"2024-02-13T14:50:12.708651+0000","last_became_active":"2024-02-13T14:49:43.982528+0000","last_became_peered":"2024-02-13T14:49:43.982528+0000","last_unstale":"2024-02-13T14:50:12.708651+0000","last_undegraded":"2024-02-13T14:50:12.708651+0000","last_fullsized":"2024-02-13T14:50:12.708651+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:47:38.808484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:14.808693+0000","last_change":"2024-02-13T14:49:30.308712+0000","last_active":"2024-02-13T14:50:14.808693+0000","last_peered":"2024-02-13T14:50:14.808693+0000","last_clean":"2024-02-13T14:50:14.808693+0000","last_became_active":"2024-02-13T14:49:30.308454+0000","last_became_peered":"2024-02-13T14:49:30.308454+0000","last_unstale":"2024-02-13T14:50:14.808693+0000","last_undegraded":"2024-02-13T14:50:14.808693+0000","last_fullsized":"2024-02-13T14:50:14.808693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:29.292511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:29.292511+0000","last_clean_scrub_stamp":"2024-02-13T14:49:29.292511+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:10:17.580074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708877+0000","last_change":"2024-02-13T14:49:48.020032+0000","last_active":"2024-02-13T14:50:12.708877+0000","last_peered":"2024-02-13T14:50:12.708877+0000","last_clean":"2024-02-13T14:50:12.708877+0000","last_became_active":"2024-02-13T14:49:48.019675+0000","last_became_peered":"2024-02-13T14:49:48.019675+0000","last_unstale":"2024-02-13T14:50:12.708877+0000","last_undegraded":"2024-02-13T14:50:12.708877+0000","last_fullsized":"2024-02-13T14:50:12.708877+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:45:43.794174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710459+0000","last_change":"2024-02-13T14:49:50.030107+0000","last_active":"2024-02-13T14:50:12.710459+0000","last_peered":"2024-02-13T14:50:12.710459+0000","last_clean":"2024-02-13T14:50:12.710459+0000","last_became_active":"2024-02-13T14:49:50.029703+0000","last_became_peered":"2024-02-13T14:49:50.029703+0000","last_unstale":"2024-02-13T14:50:12.710459+0000","last_undegraded":"2024-02-13T14:50:12.710459+0000","last_fullsized":"2024-02-13T14:50:12.710459+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:45:18.165088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706950+0000","last_change":"2024-02-13T14:49:45.999758+0000","last_active":"2024-02-13T14:50:12.706950+0000","last_peered":"2024-02-13T14:50:12.706950+0000","last_clean":"2024-02-13T14:50:12.706950+0000","last_became_active":"2024-02-13T14:49:45.999258+0000","last_became_peered":"2024-02-13T14:49:45.999258+0000","last_unstale":"2024-02-13T14:50:12.706950+0000","last_undegraded":"2024-02-13T14:50:12.706950+0000","last_fullsized":"2024-02-13T14:50:12.706950+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:53:52.999026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707415+0000","last_change":"2024-02-13T14:49:43.986159+0000","last_active":"2024-02-13T14:50:12.707415+0000","last_peered":"2024-02-13T14:50:12.707415+0000","last_clean":"2024-02-13T14:50:12.707415+0000","last_became_active":"2024-02-13T14:49:43.985652+0000","last_became_peered":"2024-02-13T14:49:43.985652+0000","last_unstale":"2024-02-13T14:50:12.707415+0000","last_undegraded":"2024-02-13T14:50:12.707415+0000","last_fullsized":"2024-02-13T14:50:12.707415+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:28:59.940724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710904+0000","last_change":"2024-02-13T14:49:48.016101+0000","last_active":"2024-02-13T14:50:12.710904+0000","last_peered":"2024-02-13T14:50:12.710904+0000","last_clean":"2024-02-13T14:50:12.710904+0000","last_became_active":"2024-02-13T14:49:48.015684+0000","last_became_peered":"2024-02-13T14:49:48.015684+0000","last_unstale":"2024-02-13T14:50:12.710904+0000","last_undegraded":"2024-02-13T14:50:12.710904+0000","last_fullsized":"2024-02-13T14:50:12.710904+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T21:05:14.129159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708138+0000","last_change":"2024-02-13T14:49:50.030492+0000","last_active":"2024-02-13T14:50:12.708138+0000","last_peered":"2024-02-13T14:50:12.708138+0000","last_clean":"2024-02-13T14:50:12.708138+0000","last_became_active":"2024-02-13T14:49:50.029776+0000","last_became_peered":"2024-02-13T14:49:50.029776+0000","last_unstale":"2024-02-13T14:50:12.708138+0000","last_undegraded":"2024-02-13T14:50:12.708138+0000","last_fullsized":"2024-02-13T14:50:12.708138+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:24:42.344672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713014+0000","last_change":"2024-02-13T14:49:43.989536+0000","last_active":"2024-02-13T14:50:12.713014+0000","last_peered":"2024-02-13T14:50:12.713014+0000","last_clean":"2024-02-13T14:50:12.713014+0000","last_became_active":"2024-02-13T14:49:43.988725+0000","last_became_peered":"2024-02-13T14:49:43.988725+0000","last_unstale":"2024-02-13T14:50:12.713014+0000","last_undegraded":"2024-02-13T14:50:12.713014+0000","last_fullsized":"2024-02-13T14:50:12.713014+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:21:02.435472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:24.037769+0000","last_change":"2024-02-13T14:49:46.000818+0000","last_active":"2024-02-13T14:50:24.037769+0000","last_peered":"2024-02-13T14:50:24.037769+0000","last_clean":"2024-02-13T14:50:24.037769+0000","last_became_active":"2024-02-13T14:49:46.000286+0000","last_became_peered":"2024-02-13T14:49:46.000286+0000","last_unstale":"2024-02-13T14:50:24.037769+0000","last_undegraded":"2024-02-13T14:50:24.037769+0000","last_fullsized":"2024-02-13T14:50:24.037769+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:11:56.998741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:24.038175+0000","last_change":"2024-02-13T14:49:48.015568+0000","last_active":"2024-02-13T14:50:24.038175+0000","last_peered":"2024-02-13T14:50:24.038175+0000","last_clean":"2024-02-13T14:50:24.038175+0000","last_became_active":"2024-02-13T14:49:48.015251+0000","last_became_peered":"2024-02-13T14:49:48.015251+0000","last_unstale":"2024-02-13T14:50:24.038175+0000","last_undegraded":"2024-02-13T14:50:24.038175+0000","last_fullsized":"2024-02-13T14:50:24.038175+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:20:31.800839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707437+0000","last_change":"2024-02-13T14:49:50.032427+0000","last_active":"2024-02-13T14:50:12.707437+0000","last_peered":"2024-02-13T14:50:12.707437+0000","last_clean":"2024-02-13T14:50:12.707437+0000","last_became_active":"2024-02-13T14:49:50.031984+0000","last_became_peered":"2024-02-13T14:49:50.031984+0000","last_unstale":"2024-02-13T14:50:12.707437+0000","last_undegraded":"2024-02-13T14:50:12.707437+0000","last_fullsized":"2024-02-13T14:50:12.707437+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:29:36.818832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709136+0000","last_change":"2024-02-13T14:49:43.990930+0000","last_active":"2024-02-13T14:50:12.709136+0000","last_peered":"2024-02-13T14:50:12.709136+0000","last_clean":"2024-02-13T14:50:12.709136+0000","last_became_active":"2024-02-13T14:49:43.990386+0000","last_became_peered":"2024-02-13T14:49:43.990386+0000","last_unstale":"2024-02-13T14:50:12.709136+0000","last_undegraded":"2024-02-13T14:50:12.709136+0000","last_fullsized":"2024-02-13T14:50:12.709136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:28:33.949170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710545+0000","last_change":"2024-02-13T14:49:46.003030+0000","last_active":"2024-02-13T14:50:12.710545+0000","last_peered":"2024-02-13T14:50:12.710545+0000","last_clean":"2024-02-13T14:50:12.710545+0000","last_became_active":"2024-02-13T14:49:46.002730+0000","last_became_peered":"2024-02-13T14:49:46.002730+0000","last_unstale":"2024-02-13T14:50:12.710545+0000","last_undegraded":"2024-02-13T14:50:12.710545+0000","last_fullsized":"2024-02-13T14:50:12.710545+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:03:36.703793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:24.037776+0000","last_change":"2024-02-13T14:49:48.016665+0000","last_active":"2024-02-13T14:50:24.037776+0000","last_peered":"2024-02-13T14:50:24.037776+0000","last_clean":"2024-02-13T14:50:24.037776+0000","last_became_active":"2024-02-13T14:49:48.016336+0000","last_became_peered":"2024-02-13T14:49:48.016336+0000","last_unstale":"2024-02-13T14:50:24.037776+0000","last_undegraded":"2024-02-13T14:50:24.037776+0000","last_fullsized":"2024-02-13T14:50:24.037776+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:23:05.130567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711081+0000","last_change":"2024-02-13T14:49:50.037038+0000","last_active":"2024-02-13T14:50:12.711081+0000","last_peered":"2024-02-13T14:50:12.711081+0000","last_clean":"2024-02-13T14:50:12.711081+0000","last_became_active":"2024-02-13T14:49:50.036715+0000","last_became_peered":"2024-02-13T14:49:50.036715+0000","last_unstale":"2024-02-13T14:50:12.711081+0000","last_undegraded":"2024-02-13T14:50:12.711081+0000","last_fullsized":"2024-02-13T14:50:12.711081+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:00:49.715548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710837+0000","last_change":"2024-02-13T14:49:43.983241+0000","last_active":"2024-02-13T14:50:12.710837+0000","last_peered":"2024-02-13T14:50:12.710837+0000","last_clean":"2024-02-13T14:50:12.710837+0000","last_became_active":"2024-02-13T14:49:43.982897+0000","last_became_peered":"2024-02-13T14:49:43.982897+0000","last_unstale":"2024-02-13T14:50:12.710837+0000","last_undegraded":"2024-02-13T14:50:12.710837+0000","last_fullsized":"2024-02-13T14:50:12.710837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:00:51.552200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708542+0000","last_change":"2024-02-13T14:49:46.001842+0000","last_active":"2024-02-13T14:50:12.708542+0000","last_peered":"2024-02-13T14:50:12.708542+0000","last_clean":"2024-02-13T14:50:12.708542+0000","last_became_active":"2024-02-13T14:49:46.001528+0000","last_became_peered":"2024-02-13T14:49:46.001528+0000","last_unstale":"2024-02-13T14:50:12.708542+0000","last_undegraded":"2024-02-13T14:50:12.708542+0000","last_fullsized":"2024-02-13T14:50:12.708542+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:02:52.234290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709583+0000","last_change":"2024-02-13T14:49:48.016738+0000","last_active":"2024-02-13T14:50:12.709583+0000","last_peered":"2024-02-13T14:50:12.709583+0000","last_clean":"2024-02-13T14:50:12.709583+0000","last_became_active":"2024-02-13T14:49:48.016423+0000","last_became_peered":"2024-02-13T14:49:48.016423+0000","last_unstale":"2024-02-13T14:50:12.709583+0000","last_undegraded":"2024-02-13T14:50:12.709583+0000","last_fullsized":"2024-02-13T14:50:12.709583+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:29:16.487976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710578+0000","last_change":"2024-02-13T14:49:50.030242+0000","last_active":"2024-02-13T14:50:12.710578+0000","last_peered":"2024-02-13T14:50:12.710578+0000","last_clean":"2024-02-13T14:50:12.710578+0000","last_became_active":"2024-02-13T14:49:50.029912+0000","last_became_peered":"2024-02-13T14:49:50.029912+0000","last_unstale":"2024-02-13T14:50:12.710578+0000","last_undegraded":"2024-02-13T14:50:12.710578+0000","last_fullsized":"2024-02-13T14:50:12.710578+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T20:45:17.200418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:24.038169+0000","last_change":"2024-02-13T14:49:48.021335+0000","last_active":"2024-02-13T14:50:24.038169+0000","last_peered":"2024-02-13T14:50:24.038169+0000","last_clean":"2024-02-13T14:50:24.038169+0000","last_became_active":"2024-02-13T14:49:48.021068+0000","last_became_peered":"2024-02-13T14:49:48.021068+0000","last_unstale":"2024-02-13T14:50:24.038169+0000","last_undegraded":"2024-02-13T14:50:24.038169+0000","last_fullsized":"2024-02-13T14:50:24.038169+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:23:18.068746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707527+0000","last_change":"2024-02-13T14:49:43.986352+0000","last_active":"2024-02-13T14:50:12.707527+0000","last_peered":"2024-02-13T14:50:12.707527+0000","last_clean":"2024-02-13T14:50:12.707527+0000","last_became_active":"2024-02-13T14:49:43.985862+0000","last_became_peered":"2024-02-13T14:49:43.985862+0000","last_unstale":"2024-02-13T14:50:12.707527+0000","last_undegraded":"2024-02-13T14:50:12.707527+0000","last_fullsized":"2024-02-13T14:50:12.707527+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:36:34.831365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709896+0000","last_change":"2024-02-13T14:49:46.002371+0000","last_active":"2024-02-13T14:50:12.709896+0000","last_peered":"2024-02-13T14:50:12.709896+0000","last_clean":"2024-02-13T14:50:12.709896+0000","last_became_active":"2024-02-13T14:49:46.002049+0000","last_became_peered":"2024-02-13T14:49:46.002049+0000","last_unstale":"2024-02-13T14:50:12.709896+0000","last_undegraded":"2024-02-13T14:50:12.709896+0000","last_fullsized":"2024-02-13T14:50:12.709896+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:37:52.548790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711464+0000","last_change":"2024-02-13T14:49:50.037137+0000","last_active":"2024-02-13T14:50:12.711464+0000","last_peered":"2024-02-13T14:50:12.711464+0000","last_clean":"2024-02-13T14:50:12.711464+0000","last_became_active":"2024-02-13T14:49:50.036413+0000","last_became_peered":"2024-02-13T14:49:50.036413+0000","last_unstale":"2024-02-13T14:50:12.711464+0000","last_undegraded":"2024-02-13T14:50:12.711464+0000","last_fullsized":"2024-02-13T14:50:12.711464+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:31:25.953981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710011+0000","last_change":"2024-02-13T14:49:48.019074+0000","last_active":"2024-02-13T14:50:12.710011+0000","last_peered":"2024-02-13T14:50:12.710011+0000","last_clean":"2024-02-13T14:50:12.710011+0000","last_became_active":"2024-02-13T14:49:48.018654+0000","last_became_peered":"2024-02-13T14:49:48.018654+0000","last_unstale":"2024-02-13T14:50:12.710011+0000","last_undegraded":"2024-02-13T14:50:12.710011+0000","last_fullsized":"2024-02-13T14:50:12.710011+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:43:11.627486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706734+0000","last_change":"2024-02-13T14:49:43.990111+0000","last_active":"2024-02-13T14:50:12.706734+0000","last_peered":"2024-02-13T14:50:12.706734+0000","last_clean":"2024-02-13T14:50:12.706734+0000","last_became_active":"2024-02-13T14:49:43.989832+0000","last_became_peered":"2024-02-13T14:49:43.989832+0000","last_unstale":"2024-02-13T14:50:12.706734+0000","last_undegraded":"2024-02-13T14:50:12.706734+0000","last_fullsized":"2024-02-13T14:50:12.706734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T14:50:27.435987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709677+0000","last_change":"2024-02-13T14:49:45.998559+0000","last_active":"2024-02-13T14:50:12.709677+0000","last_peered":"2024-02-13T14:50:12.709677+0000","last_clean":"2024-02-13T14:50:12.709677+0000","last_became_active":"2024-02-13T14:49:45.998339+0000","last_became_peered":"2024-02-13T14:49:45.998339+0000","last_unstale":"2024-02-13T14:50:12.709677+0000","last_undegraded":"2024-02-13T14:50:12.709677+0000","last_fullsized":"2024-02-13T14:50:12.709677+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T14:59:48.384896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710057+0000","last_change":"2024-02-13T14:49:50.037217+0000","last_active":"2024-02-13T14:50:12.710057+0000","last_peered":"2024-02-13T14:50:12.710057+0000","last_clean":"2024-02-13T14:50:12.710057+0000","last_became_active":"2024-02-13T14:49:50.036855+0000","last_became_peered":"2024-02-13T14:49:50.036855+0000","last_unstale":"2024-02-13T14:50:12.710057+0000","last_undegraded":"2024-02-13T14:50:12.710057+0000","last_fullsized":"2024-02-13T14:50:12.710057+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:47:14.570295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711268+0000","last_change":"2024-02-13T14:49:43.984120+0000","last_active":"2024-02-13T14:50:12.711268+0000","last_peered":"2024-02-13T14:50:12.711268+0000","last_clean":"2024-02-13T14:50:12.711268+0000","last_became_active":"2024-02-13T14:49:43.983801+0000","last_became_peered":"2024-02-13T14:49:43.983801+0000","last_unstale":"2024-02-13T14:50:12.711268+0000","last_undegraded":"2024-02-13T14:50:12.711268+0000","last_fullsized":"2024-02-13T14:50:12.711268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:07:01.039824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709239+0000","last_change":"2024-02-13T14:49:45.999416+0000","last_active":"2024-02-13T14:50:12.709239+0000","last_peered":"2024-02-13T14:50:12.709239+0000","last_clean":"2024-02-13T14:50:12.709239+0000","last_became_active":"2024-02-13T14:49:45.998986+0000","last_became_peered":"2024-02-13T14:49:45.998986+0000","last_unstale":"2024-02-13T14:50:12.709239+0000","last_undegraded":"2024-02-13T14:50:12.709239+0000","last_fullsized":"2024-02-13T14:50:12.709239+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T16:44:43.288856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709302+0000","last_change":"2024-02-13T14:49:48.019375+0000","last_active":"2024-02-13T14:50:12.709302+0000","last_peered":"2024-02-13T14:50:12.709302+0000","last_clean":"2024-02-13T14:50:12.709302+0000","last_became_active":"2024-02-13T14:49:48.019054+0000","last_became_peered":"2024-02-13T14:49:48.019054+0000","last_unstale":"2024-02-13T14:50:12.709302+0000","last_undegraded":"2024-02-13T14:50:12.709302+0000","last_fullsized":"2024-02-13T14:50:12.709302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:41:22.217181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707804+0000","last_change":"2024-02-13T14:49:50.034507+0000","last_active":"2024-02-13T14:50:12.707804+0000","last_peered":"2024-02-13T14:50:12.707804+0000","last_clean":"2024-02-13T14:50:12.707804+0000","last_became_active":"2024-02-13T14:49:50.033952+0000","last_became_peered":"2024-02-13T14:49:50.033952+0000","last_unstale":"2024-02-13T14:50:12.707804+0000","last_undegraded":"2024-02-13T14:50:12.707804+0000","last_fullsized":"2024-02-13T14:50:12.707804+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:07:54.700777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:24.038186+0000","last_change":"2024-02-13T14:49:48.021205+0000","last_active":"2024-02-13T14:50:24.038186+0000","last_peered":"2024-02-13T14:50:24.038186+0000","last_clean":"2024-02-13T14:50:24.038186+0000","last_became_active":"2024-02-13T14:49:48.020866+0000","last_became_peered":"2024-02-13T14:49:48.020866+0000","last_unstale":"2024-02-13T14:50:24.038186+0000","last_undegraded":"2024-02-13T14:50:24.038186+0000","last_fullsized":"2024-02-13T14:50:24.038186+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:30:53.921682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.713179+0000","last_change":"2024-02-13T14:49:43.988031+0000","last_active":"2024-02-13T14:50:12.713179+0000","last_peered":"2024-02-13T14:50:12.713179+0000","last_clean":"2024-02-13T14:50:12.713179+0000","last_became_active":"2024-02-13T14:49:43.987483+0000","last_became_peered":"2024-02-13T14:49:43.987483+0000","last_unstale":"2024-02-13T14:50:12.713179+0000","last_undegraded":"2024-02-13T14:50:12.713179+0000","last_fullsized":"2024-02-13T14:50:12.713179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:37:37.979522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708698+0000","last_change":"2024-02-13T14:49:46.001560+0000","last_active":"2024-02-13T14:50:12.708698+0000","last_peered":"2024-02-13T14:50:12.708698+0000","last_clean":"2024-02-13T14:50:12.708698+0000","last_became_active":"2024-02-13T14:49:46.001003+0000","last_became_peered":"2024-02-13T14:49:46.001003+0000","last_unstale":"2024-02-13T14:50:12.708698+0000","last_undegraded":"2024-02-13T14:50:12.708698+0000","last_fullsized":"2024-02-13T14:50:12.708698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:04:16.957090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710241+0000","last_change":"2024-02-13T14:49:50.037090+0000","last_active":"2024-02-13T14:50:12.710241+0000","last_peered":"2024-02-13T14:50:12.710241+0000","last_clean":"2024-02-13T14:50:12.710241+0000","last_became_active":"2024-02-13T14:49:50.036712+0000","last_became_peered":"2024-02-13T14:49:50.036712+0000","last_unstale":"2024-02-13T14:50:12.710241+0000","last_undegraded":"2024-02-13T14:50:12.710241+0000","last_fullsized":"2024-02-13T14:50:12.710241+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:54:55.701678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711656+0000","last_change":"2024-02-13T14:49:43.985406+0000","last_active":"2024-02-13T14:50:12.711656+0000","last_peered":"2024-02-13T14:50:12.711656+0000","last_clean":"2024-02-13T14:50:12.711656+0000","last_became_active":"2024-02-13T14:49:43.984553+0000","last_became_peered":"2024-02-13T14:49:43.984553+0000","last_unstale":"2024-02-13T14:50:12.711656+0000","last_undegraded":"2024-02-13T14:50:12.711656+0000","last_fullsized":"2024-02-13T14:50:12.711656+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:23:58.566162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.709125+0000","last_change":"2024-02-13T14:49:46.001819+0000","last_active":"2024-02-13T14:50:12.709125+0000","last_peered":"2024-02-13T14:50:12.709125+0000","last_clean":"2024-02-13T14:50:12.709125+0000","last_became_active":"2024-02-13T14:49:46.001360+0000","last_became_peered":"2024-02-13T14:49:46.001360+0000","last_unstale":"2024-02-13T14:50:12.709125+0000","last_undegraded":"2024-02-13T14:50:12.709125+0000","last_fullsized":"2024-02-13T14:50:12.709125+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:16:26.234559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707967+0000","last_change":"2024-02-13T14:49:48.019109+0000","last_active":"2024-02-13T14:50:12.707967+0000","last_peered":"2024-02-13T14:50:12.707967+0000","last_clean":"2024-02-13T14:50:12.707967+0000","last_became_active":"2024-02-13T14:49:48.018765+0000","last_became_peered":"2024-02-13T14:49:48.018765+0000","last_unstale":"2024-02-13T14:50:12.707967+0000","last_undegraded":"2024-02-13T14:50:12.707967+0000","last_fullsized":"2024-02-13T14:50:12.707967+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:51:35.743346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.706443+0000","last_change":"2024-02-13T14:49:50.028969+0000","last_active":"2024-02-13T14:50:12.706443+0000","last_peered":"2024-02-13T14:50:12.706443+0000","last_clean":"2024-02-13T14:50:12.706443+0000","last_became_active":"2024-02-13T14:49:50.028662+0000","last_became_peered":"2024-02-13T14:49:50.028662+0000","last_unstale":"2024-02-13T14:50:12.706443+0000","last_undegraded":"2024-02-13T14:50:12.706443+0000","last_fullsized":"2024-02-13T14:50:12.706443+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T18:24:58.629098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.711676+0000","last_change":"2024-02-13T14:49:43.989780+0000","last_active":"2024-02-13T14:50:12.711676+0000","last_peered":"2024-02-13T14:50:12.711676+0000","last_clean":"2024-02-13T14:50:12.711676+0000","last_became_active":"2024-02-13T14:49:43.989245+0000","last_became_peered":"2024-02-13T14:49:43.989245+0000","last_unstale":"2024-02-13T14:50:12.711676+0000","last_undegraded":"2024-02-13T14:50:12.711676+0000","last_fullsized":"2024-02-13T14:50:12.711676+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T23:16:36.989280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710489+0000","last_change":"2024-02-13T14:49:46.002672+0000","last_active":"2024-02-13T14:50:12.710489+0000","last_peered":"2024-02-13T14:50:12.710489+0000","last_clean":"2024-02-13T14:50:12.710489+0000","last_became_active":"2024-02-13T14:49:46.002302+0000","last_became_peered":"2024-02-13T14:49:46.002302+0000","last_unstale":"2024-02-13T14:50:12.710489+0000","last_undegraded":"2024-02-13T14:50:12.710489+0000","last_fullsized":"2024-02-13T14:50:12.710489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T01:04:29.294460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707803+0000","last_change":"2024-02-13T14:49:48.017192+0000","last_active":"2024-02-13T14:50:12.707803+0000","last_peered":"2024-02-13T14:50:12.707803+0000","last_clean":"2024-02-13T14:50:12.707803+0000","last_became_active":"2024-02-13T14:49:48.016842+0000","last_became_peered":"2024-02-13T14:49:48.016842+0000","last_unstale":"2024-02-13T14:50:12.707803+0000","last_undegraded":"2024-02-13T14:50:12.707803+0000","last_fullsized":"2024-02-13T14:50:12.707803+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T02:14:45.811878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.707506+0000","last_change":"2024-02-13T14:49:50.030701+0000","last_active":"2024-02-13T14:50:12.707506+0000","last_peered":"2024-02-13T14:50:12.707506+0000","last_clean":"2024-02-13T14:50:12.707506+0000","last_became_active":"2024-02-13T14:49:50.030248+0000","last_became_peered":"2024-02-13T14:49:50.030248+0000","last_unstale":"2024-02-13T14:50:12.707506+0000","last_undegraded":"2024-02-13T14:50:12.707506+0000","last_fullsized":"2024-02-13T14:50:12.707506+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:49.011808+0000","last_clean_scrub_stamp":"2024-02-13T14:49:49.011808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T22:42:32.167292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.708820+0000","last_change":"2024-02-13T14:49:43.990756+0000","last_active":"2024-02-13T14:50:12.708820+0000","last_peered":"2024-02-13T14:50:12.708820+0000","last_clean":"2024-02-13T14:50:12.708820+0000","last_became_active":"2024-02-13T14:49:43.990309+0000","last_became_peered":"2024-02-13T14:49:43.990309+0000","last_unstale":"2024-02-13T14:50:12.708820+0000","last_undegraded":"2024-02-13T14:50:12.708820+0000","last_fullsized":"2024-02-13T14:50:12.708820+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:42.966484+0000","last_clean_scrub_stamp":"2024-02-13T14:49:42.966484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T00:40:42.636185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710896+0000","last_change":"2024-02-13T14:49:46.001636+0000","last_active":"2024-02-13T14:50:12.710896+0000","last_peered":"2024-02-13T14:50:12.710896+0000","last_clean":"2024-02-13T14:50:12.710896+0000","last_became_active":"2024-02-13T14:49:46.001183+0000","last_became_peered":"2024-02-13T14:49:46.001183+0000","last_unstale":"2024-02-13T14:50:12.710896+0000","last_undegraded":"2024-02-13T14:50:12.710896+0000","last_fullsized":"2024-02-13T14:50:12.710896+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:44.982556+0000","last_clean_scrub_stamp":"2024-02-13T14:49:44.982556+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T15:20:32.476479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-13T14:50:12.710144+0000","last_change":"2024-02-13T14:49:48.017031+0000","last_active":"2024-02-13T14:50:12.710144+0000","last_peered":"2024-02-13T14:50:12.710144+0000","last_clean":"2024-02-13T14:50:12.710144+0000","last_became_active":"2024-02-13T14:49:48.016712+0000","last_became_peered":"2024-02-13T14:49:48.016712+0000","last_unstale":"2024-02-13T14:50:12.710144+0000","last_undegraded":"2024-02-13T14:50:12.710144+0000","last_fullsized":"2024-02-13T14:50:12.710144+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-13T14:49:46.995634+0000","last_clean_scrub_stamp":"2024-02-13T14:49:46.995634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-14T19:54:23.016529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411344,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 14:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.52500000000000002}]},{"osd":1,"last update":"Tue Feb 13 14:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.38100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738392,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 13 14:50:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.70599999999999996}]},{"osd":2,"last update":"Tue Feb 13 14:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.66500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 13 14:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.38700000000000001}]},{"osd":2,"last update":"Tue Feb 13 14:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-13T14:50:38.870 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-13T14:50:38.870 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-13T14:50:38.870 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-13T14:50:38.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- ceph health --format=json 2024-02-13T14:50:38.955 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:38 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:39.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:38 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:39.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:38 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:39.379 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:39.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:39 smithi103 ceph-mon[25851]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:40.079 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:39 smithi158 ceph-mon[27628]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:40.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:39 smithi164 ceph-mon[27636]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:42.238 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-13T14:50:42.238 INFO:teuthology.orchestra.run.smithi103.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-13T14:50:42.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:42 smithi164 ceph-mon[27636]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:42.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:42 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:42 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:42 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:42 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:42 smithi158 ceph-mon[27628]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:42.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:42 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:42 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:42 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:42 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.627 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:42 smithi103 ceph-mon[25851]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:42.627 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:42 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.627 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:42 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.627 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:42 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.627 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:42 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:42.913 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-13T14:50:42.913 INFO:tasks.cephadm:Setup complete, yielding 2024-02-13T14:50:42.914 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-13T14:50:42.925 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi103.front.sepia.ceph.com 2024-02-13T14:50:42.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-02-13T14:50:43.357 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:43.485 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:43 smithi103 ceph-mon[25851]: from='client.? 172.21.15.103:0/3852660653' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-13T14:50:43.485 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:43 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:43.485 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:43 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:43.485 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:43 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:43.485 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:43 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:43.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:43 smithi164 ceph-mon[27636]: from='client.? 172.21.15.103:0/3852660653' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-13T14:50:43.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:43 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:43.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:43 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:43.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:43 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:43.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:43 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:43.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:43 smithi158 ceph-mon[27628]: from='client.? 172.21.15.103:0/3852660653' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-13T14:50:43.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:43 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:43.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:43 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:43.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:43 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:43.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:43 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:43.808 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:50:43 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[48910]: ts=2024-02-13T14:50:43.599Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004245798s 2024-02-13T14:50:44.674 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:44 smithi103 ceph-mon[25851]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:44.674 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:44 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:44.674 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:44 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:44.675 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:44 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:44.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:44 smithi158 ceph-mon[27628]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:44.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:44 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:44.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:44 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:44.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:44 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:44.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:44 smithi164 ceph-mon[27636]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:44.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:44 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:44.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:44 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:44.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:44 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:45.014 INFO:teuthology.orchestra.run.smithi103.stdout:167 167 2024-02-13T14:50:45.267 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:45 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ::ffff:172.21.15.158 - - [13/Feb/2024:14:50:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-13T14:50:45.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- bash -c 'ceph orch status' 2024-02-13T14:50:45.838 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:46.533 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:46 smithi103 ceph-mon[25851]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:46.645 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:46 smithi158 ceph-mon[27628]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:46.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:46 smithi164 ceph-mon[27636]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:47.569 INFO:teuthology.orchestra.run.smithi103.stdout:Backend: cephadm 2024-02-13T14:50:47.569 INFO:teuthology.orchestra.run.smithi103.stdout:Available: Yes 2024-02-13T14:50:47.569 INFO:teuthology.orchestra.run.smithi103.stdout:Paused: No 2024-02-13T14:50:48.341 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-13T14:50:48.422 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:48 smithi158 ceph-mon[27628]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:48.422 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:48 smithi158 ceph-mon[27628]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:50:48.583 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:48.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:48 smithi103 ceph-mon[25851]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:48.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:48 smithi103 ceph-mon[25851]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:50:48.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:48 smithi164 ceph-mon[27636]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:48.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:48 smithi164 ceph-mon[27636]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:50:49.100 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:48 smithi158 systemd[1]: Starting Ceph grafana.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:50:49.368 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 podman[38226]: 2024-02-13 14:50:49.097898547 +0000 UTC m=+0.112381045 container create 9077e6c625be1f6183243007cbb6947b61ad40a15dda8c9227d92a150c334229 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a, maintainer=Paul Cuzner , io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., vcs-type=git, version=8.5, name=ubi8, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, description=Ceph Grafana Container, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-02-13T14:50:49.368 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 podman[38226]: 2024-02-13 14:50:49.002583187 +0000 UTC m=+0.017065695 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-02-13T14:50:49.618 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 podman[38226]: 2024-02-13 14:50:49.366729624 +0000 UTC m=+0.381212130 container init 9077e6c625be1f6183243007cbb6947b61ad40a15dda8c9227d92a150c334229 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a, vcs-type=git, version=8.5, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, name=ubi8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, release=239.1651231664, description=Ceph Grafana Container) 2024-02-13T14:50:49.619 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 podman[38226]: 2024-02-13 14:50:49.372086216 +0000 UTC m=+0.386568712 container start 9077e6c625be1f6183243007cbb6947b61ad40a15dda8c9227d92a150c334229 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, maintainer=Paul Cuzner , release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, distribution-scope=public, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, version=8.5) 2024-02-13T14:50:49.619 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 bash[38226]: 9077e6c625be1f6183243007cbb6947b61ad40a15dda8c9227d92a150c334229 2024-02-13T14:50:49.619 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 systemd[1]: Started Ceph grafana.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:50:49.619 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534217819Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-02-13T14:50:49.619 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534537856Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-02-13T14:50:49.619 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534565083Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-02-13T14:50:49.619 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534583243Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-02-13T14:50:49.619 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534594632Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-13T14:50:49.619 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534605653Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-13T14:50:49.619 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534616367Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-13T14:50:49.619 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534628057Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534639254Z level=info msg="Path Home" path=/usr/share/grafana 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534649875Z level=info msg="Path Data" path=/var/lib/grafana 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534660338Z level=info msg="Path Logs" path=/var/log/grafana 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534670548Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534681154Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=settings t=2024-02-13T14:50:49.534691753Z level=info msg="App mode production" 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=sqlstore t=2024-02-13T14:50:49.534756103Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=sqlstore t=2024-02-13T14:50:49.534786065Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.536135627Z level=info msg="Starting DB migrations" 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.53625488Z level=info msg="Executing migration" id="create migration_log table" 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.536966872Z level=info msg="Executing migration" id="create user table" 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.537535397Z level=info msg="Executing migration" id="add unique index user.login" 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.538070455Z level=info msg="Executing migration" id="add unique index user.email" 2024-02-13T14:50:49.620 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.538534887Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-02-13T14:50:49.621 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.539264654Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-02-13T14:50:49.622 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.539768533Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-02-13T14:50:49.622 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.5407812Z level=info msg="Executing migration" id="create user table v2" 2024-02-13T14:50:49.622 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.541342182Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-02-13T14:50:49.622 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.541823254Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-02-13T14:50:49.622 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.542276382Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-02-13T14:50:49.622 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.542583792Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-02-13T14:50:49.622 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.542888488Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-02-13T14:50:49.622 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.543282581Z level=info msg="Executing migration" id="Update user table charset" 2024-02-13T14:50:49.622 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.543519772Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-02-13T14:50:49.622 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.54387118Z level=info msg="Executing migration" id="Add missing user data" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.544211521Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.544564282Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.544905234Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.545320544Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.54644434Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.546878601Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.547308502Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.547715088Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.54812123Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.548511483Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.548760911Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.549120506Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.549681261Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.550082849Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.550461075Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-13T14:50:49.623 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.55110087Z level=info msg="Executing migration" id="create temp_user v2" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.551472913Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.551828493Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.552242807Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.552597907Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.553032128Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.553343934Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.553642946Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.553977179Z level=info msg="Executing migration" id="create star table" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.554320008Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.554739611Z level=info msg="Executing migration" id="create org table v1" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.55517168Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.555586314Z level=info msg="Executing migration" id="create org_user table v1" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.555996672Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.556405605Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-02-13T14:50:49.624 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.556814605Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.557218749Z level=info msg="Executing migration" id="Update org table charset" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.557469582Z level=info msg="Executing migration" id="Update org_user table charset" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.557727734Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.558095965Z level=info msg="Executing migration" id="create dashboard table" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.5585015Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.558894746Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.559303648Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.55975297Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.56019703Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.560599968Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.561454692Z level=info msg="Executing migration" id="create dashboard v2" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.56187116Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.562310124Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.56274495Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.563086218Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-02-13T14:50:49.625 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.563464475Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.563729957Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.564219342Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.564686844Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.565158917Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.565570813Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.566050985Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.566469342Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.566847561Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.567123936Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.567356012Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.56782981Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.568481384Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.568984466Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.569471263Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.569765359Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-02-13T14:50:49.626 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.570184965Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.570645385Z level=info msg="Executing migration" id="Update dashboard title length" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.571021768Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.571482445Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.571897798Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.572764363Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.573142639Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.573544807Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.57395879Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.574283027Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.574641206Z level=info msg="Executing migration" id="Add check_sum column" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.575139662Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.575534585Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.575787977Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.576281137Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-02-13T14:50:49.627 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.576689758Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.577371876Z level=info msg="Executing migration" id="create data_source table" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.577919368Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.57846499Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.579051477Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.579572489Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.580088141Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.581672257Z level=info msg="Executing migration" id="create data_source table v2" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.58228427Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.582835875Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.58339341Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.583823641Z level=info msg="Executing migration" id="Add column with_credentials" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.584570948Z level=info msg="Executing migration" id="Add secure json data column" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.585337563Z level=info msg="Executing migration" id="Update data_source table charset" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.585707028Z level=info msg="Executing migration" id="Update initial version to 1" 2024-02-13T14:50:49.628 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.586112759Z level=info msg="Executing migration" id="Add read_only data column" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.586739744Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.587075504Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.588746462Z level=info msg="Executing migration" id="Add uid column" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.589424229Z level=info msg="Executing migration" id="Update uid value" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.58968925Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.590078277Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.590468204Z level=info msg="Executing migration" id="create api_key table" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.590832319Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.591253061Z level=info msg="Executing migration" id="add index api_key.key" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.592115863Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.59280522Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.593463629Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.593808981Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.594188544Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-02-13T14:50:49.629 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.595208531Z level=info msg="Executing migration" id="create api_key table v2" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.595565544Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.595917427Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.596293161Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.596687129Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.597018461Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.597338588Z level=info msg="Executing migration" id="Update api_key table charset" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.597568841Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.598084193Z level=info msg="Executing migration" id="Add service account foreign key" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.5985951Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.598850018Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.599348733Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.599828193Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.60018845Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.600486649Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.600808241Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.601201472Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-13T14:50:49.630 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.601573649Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-13T14:50:49.631 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.601948786Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-02-13T14:50:49.631 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.602217538Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-02-13T14:50:49.631 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.602434359Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-02-13T14:50:49.631 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.60295712Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-02-13T14:50:49.631 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.603465866Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-13T14:50:49.631 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.603766312Z level=info msg="Executing migration" id="create quota table v1" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.604129945Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.604522103Z level=info msg="Executing migration" id="Update quota table charset" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.604762349Z level=info msg="Executing migration" id="create plugin_setting table" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.605126149Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.605505371Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.606064845Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.606364951Z level=info msg="Executing migration" id="create session table" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.606770675Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.607086897Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.607367946Z level=info msg="Executing migration" id="create playlist table v2" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.607733059Z level=info msg="Executing migration" id="create playlist item table v2" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.608096689Z level=info msg="Executing migration" id="Update playlist table charset" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.608363219Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-02-13T14:50:49.632 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.608599814Z level=info msg="Executing migration" id="drop preferences table v2" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.60884465Z level=info msg="Executing migration" id="drop preferences table v3" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.60913303Z level=info msg="Executing migration" id="create preferences table v3" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.609487663Z level=info msg="Executing migration" id="Update preferences table charset" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.609733473Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.61036669Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.610660343Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.611257652Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.611817177Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.612106422Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.612497439Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.612924054Z level=info msg="Executing migration" id="create alert table v1" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.613365512Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.613784895Z level=info msg="Executing migration" id="add index alert state" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.614171408Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.614545344Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-02-13T14:50:49.633 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.614892883Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-13T14:50:49.634 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.615303261Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-13T14:50:49.634 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.615635458Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-13T14:50:49.634 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.616986558Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-02-13T14:50:49.634 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.617444043Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-13T14:50:49.634 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.618008748Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-02-13T14:50:49.634 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.618416442Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-02-13T14:50:49.634 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.618777296Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-02-13T14:50:49.634 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.619158245Z level=info msg="Executing migration" id="Add column is_default" 2024-02-13T14:50:49.634 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.620046459Z level=info msg="Executing migration" id="Add column frequency" 2024-02-13T14:50:49.634 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.620925464Z level=info msg="Executing migration" id="Add column send_reminder" 2024-02-13T14:50:49.634 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.621537535Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-02-13T14:50:49.877 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.622155489Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.622550009Z level=info msg="Executing migration" id="Update alert table charset" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.622771673Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.623029395Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.623385246Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.623768278Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.624128183Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.624488283Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.624897213Z level=info msg="Executing migration" id="Add for to alert table" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.625537034Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.627146961Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.627531127Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.628083001Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-02-13T14:50:49.878 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.628593792Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-02-13T14:50:49.879 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.629251647Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-02-13T14:50:49.879 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.629518312Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-02-13T14:50:49.879 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.629882084Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-02-13T14:50:49.879 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.630263736Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-02-13T14:50:49.880 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.630524997Z level=info msg="Executing migration" id="create annotation table v5" 2024-02-13T14:50:49.880 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.630920025Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-02-13T14:50:49.880 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.631301307Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-02-13T14:50:49.880 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.631669114Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-02-13T14:50:49.880 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.632104001Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-02-13T14:50:49.880 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.632505557Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-02-13T14:50:49.880 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.632912044Z level=info msg="Executing migration" id="Update annotation table charset" 2024-02-13T14:50:49.880 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.633178657Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-02-13T14:50:49.880 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.633759142Z level=info msg="Executing migration" id="Drop category_id index" 2024-02-13T14:50:49.880 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.634140563Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.634751463Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.635117481Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.635501055Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.635831024Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.637361731Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.637709198Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.638107596Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.638392901Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.638675902Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.638952888Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.639708357Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.640656839Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.641222698Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-02-13T14:50:49.883 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.641643564Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.641901022Z level=info msg="Executing migration" id="Add epoch_end column" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.642530536Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.642936508Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.643220976Z level=info msg="Executing migration" id="Move region to single row" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.643512766Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.643856433Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.644238419Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.644597256Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.644990515Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.645351644Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.645688906Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.645973453Z level=info msg="Executing migration" id="create test_data table" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.646342765Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-02-13T14:50:49.884 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.646695472Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.647087606Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.647465284Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.647707134Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.648001821Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.648258369Z level=info msg="Executing migration" id="create team table" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.648601114Z level=info msg="Executing migration" id="add index team.org_id" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.649038627Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.649439728Z level=info msg="Executing migration" id="create team member table" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.649782596Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.650178562Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.650560023Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.65094977Z level=info msg="Executing migration" id="Add column email to team table" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.651635597Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-02-13T14:50:49.885 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.652582449Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.653275253Z level=info msg="Executing migration" id="create dashboard acl table" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.65365443Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.654100721Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.654511782Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.654955844Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.655426823Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.655825794Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.65625975Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.656653748Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.657056795Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.65734943Z level=info msg="Executing migration" id="create tag table" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.657700121Z level=info msg="Executing migration" id="add index tag.key_value" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.658122618Z level=info msg="Executing migration" id="create login attempt table" 2024-02-13T14:50:49.886 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.658490413Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.65889694Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.659292565Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.661133298Z level=info msg="Executing migration" id="create login_attempt v2" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.661524979Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.661940391Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.662245474Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.662555801Z level=info msg="Executing migration" id="create user auth table" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.662892461Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.663326925Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.663596863Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.664341809Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.665078776Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.665799705Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-02-13T14:50:49.887 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.666554296Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.666973997Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.667705823Z level=info msg="Executing migration" id="create server_lock table" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.668106476Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.668549437Z level=info msg="Executing migration" id="create user auth token table" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.668952469Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.669382419Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.669785695Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.670193139Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.670939943Z level=info msg="Executing migration" id="create cache_data table" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.671351491Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.671784323Z level=info msg="Executing migration" id="create short_url table v1" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.672205543Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.672609109Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-02-13T14:50:49.888 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.672855235Z level=info msg="Executing migration" id="delete alert_definition table" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.67313235Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.673539746Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.673953627Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.674385372Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.674630555Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.675036271Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.675433048Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.675823258Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.676243614Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.677016177Z level=info msg="Executing migration" id="drop alert_definition table" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.677423461Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.677682727Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-02-13T14:50:49.889 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.678067146Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-13T14:50:49.890 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.678504274Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-13T14:50:49.890 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.678894227Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-13T14:50:49.890 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.679170738Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-02-13T14:50:49.890 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.679540163Z level=info msg="Executing migration" id="create alert_instance table" 2024-02-13T14:50:49.890 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.67990445Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-13T14:50:49.890 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.680313731Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-13T14:50:49.891 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.68073647Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-02-13T14:50:49.891 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.681489213Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-13T14:50:49.891 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.68191587Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-02-13T14:50:49.891 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.682319216Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-02-13T14:50:49.891 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.685639341Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-02-13T14:50:49.891 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.688921046Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.689346437Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.689725764Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.690512911Z level=info msg="Executing migration" id="create alert_rule table" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.690925264Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.691386457Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.691849155Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.692269192Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.69250231Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.693292262Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.694065531Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.694859807Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.695252181Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.695613186Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-02-13T14:50:49.892 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.696381293Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.697152767Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.69755658Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.698335755Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.699133007Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.699415586Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.699825676Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.700263747Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.700661504Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.700908361Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.701750823Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.702543676Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.703408833Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.704272225Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.705111332Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-02-13T14:50:49.893 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.705386178Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.705776213Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.706596793Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.706859425Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.707740002Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.708184011Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.709017004Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.709383999Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.709801126Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.710628253Z level=info msg="Executing migration" id="create provenance_type table" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.711026721Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.711451031Z level=info msg="Executing migration" id="create alert_image table" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.711825174Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.712264838Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.712535009Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.712942315Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-02-13T14:50:49.894 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.713403374Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.713482207Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.713708297Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.714052838Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.714496298Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.714624521Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.714754401Z level=info msg="alerts found to migrate" alerts=0 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.715209027Z level=info msg="Executing migration" id="create library_element table v1" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.715647929Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.716085678Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.71647009Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.716905447Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.717381426Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.717622811Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-02-13T14:50:49.895 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.717990649Z level=info msg="Executing migration" id="create data_keys table" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.718410036Z level=info msg="Executing migration" id="create secrets table" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.718839575Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.724174026Z level=info msg="Executing migration" id="add name column into data_keys" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.725335481Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.725630044Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.729856433Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.734055767Z level=info msg="Executing migration" id="create kv_store table v1" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.734446551Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.734885748Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.735309704Z level=info msg="Executing migration" id="create permission table" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.73567678Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.736114274Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-02-13T14:50:49.896 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.736569039Z level=info msg="Executing migration" id="create role table" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.737035243Z level=info msg="Executing migration" id="add column display_name" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.73802016Z level=info msg="Executing migration" id="add column group_name" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.738910662Z level=info msg="Executing migration" id="add index role.org_id" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.739329163Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.73972484Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.740140159Z level=info msg="Executing migration" id="create team role table" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.740495686Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.7409147Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.741373219Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.74179381Z level=info msg="Executing migration" id="create user role table" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.742190352Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-02-13T14:50:49.897 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.74262652Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-02-13T14:50:49.898 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.743083166Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-02-13T14:50:49.898 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.743504675Z level=info msg="Executing migration" id="create builtin role table" 2024-02-13T14:50:49.898 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.743903239Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-02-13T14:50:49.898 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.744374296Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-02-13T14:50:49.898 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.744782214Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.745746002Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.746264235Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.746701306Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.747126362Z level=info msg="Executing migration" id="add unique index role.uid" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.747543747Z level=info msg="Executing migration" id="create seed assignment table" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.747905157Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.748343443Z level=info msg="Executing migration" id="add column hidden to role table" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.750494615Z level=info msg="Executing migration" id="create query_history table v1" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.750888926Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.751324841Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.751559139Z level=info msg="Executing migration" id="teams permissions migration" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.751980805Z level=info msg="Executing migration" id="dashboard permissions" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.752426883Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-02-13T14:50:49.899 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.752729038Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.753031654Z level=info msg="Executing migration" id="alerting notification permissions" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.753352824Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.753706982Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.754123331Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.755051173Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.755319196Z level=info msg="Executing migration" id="create correlation table v1" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.755692019Z level=info msg="Executing migration" id="add index correlations.uid" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.756104145Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.756515713Z level=info msg="Executing migration" id="add correlation config column" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.757524012Z level=info msg="Executing migration" id="create entity_events table" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.757928998Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.758312553Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-13T14:50:49.900 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.758393086Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.758652428Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.758735243Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.758987889Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.759373042Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.75972703Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.760140196Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.760538405Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.760907978Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.761305028Z level=info msg="Executing migration" id="Drop public config table" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.761671344Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.762192979Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.76270216Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.763197931Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-02-13T14:50:49.901 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.763599798Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.766601019Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.767585063Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.768573967Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.768883005Z level=info msg="Executing migration" id="add share column" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.769885052Z level=info msg="Executing migration" id="create default alerting folders" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.770344691Z level=info msg="Executing migration" id="create file table" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.770755433Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.771192604Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.771632828Z level=info msg="Executing migration" id="create file_meta table" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.772002827Z level=info msg="Executing migration" id="file table idx: path key" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.77241703Z level=info msg="Executing migration" id="set path collation in file table" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.772648087Z level=info msg="Executing migration" id="managed permissions migration" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.773000537Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.773289319Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-02-13T14:50:49.902 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.773649549Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.774872749Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.775205445Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.775606357Z level=info msg="Executing migration" id="update group index for alert rules" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.775966124Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.776249161Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.776548368Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.777519165Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.778501061Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.77889425Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.788588438Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.78902607Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.789454627Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.793196166Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.79348761Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-02-13T14:50:49.903 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.793773649Z level=info msg="Executing migration" id="create folder table" 2024-02-13T14:50:49.904 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.794144578Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-02-13T14:50:49.904 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.794550184Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-02-13T14:50:49.904 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.794959424Z level=info msg="Executing migration" id="Update folder title length" 2024-02-13T14:50:49.904 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.795211547Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-02-13T14:50:49.904 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=migrator t=2024-02-13T14:50:49.795594173Z level=info msg="migrations completed" performed=485 skipped=0 duration=259.378058ms 2024-02-13T14:50:49.904 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=sqlstore t=2024-02-13T14:50:49.795995559Z level=info msg="Created default organization" 2024-02-13T14:50:49.904 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=plugin.loader t=2024-02-13T14:50:49.857899344Z level=info msg="Plugin registered" pluginID=input 2024-02-13T14:50:50.178 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=plugin.loader t=2024-02-13T14:50:49.886774338Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-02-13T14:50:50.178 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=plugin.loader t=2024-02-13T14:50:49.886820977Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-02-13T14:50:50.178 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=secrets t=2024-02-13T14:50:49.886867803Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-02-13T14:50:50.178 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=query_data t=2024-02-13T14:50:49.889041655Z level=info msg="Query Service initialization" 2024-02-13T14:50:50.178 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:49 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=live.push_http t=2024-02-13T14:50:49.892322407Z level=info msg="Live Push Gateway initialization" 2024-02-13T14:50:50.557 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:agent.smithi103 smithi103 running 7s ago 3m - - 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:agent.smithi158 smithi158 running 8s ago 3m - - 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:agent.smithi164 smithi164 running 8s ago 3m - - 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.a smithi103 *:9093,9094 running (17s) 7s ago 16s 17.3M - 0.25.0 c8568f914cd2 23bbc8342d9c 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.a smithi158 *:3000 starting - - - - 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.x smithi158 *:8443,8765 running (2m) 8s ago 2m 449M - 19.0.0-1423-ge1b79d5a 28480620e5ed 1f155c69d49c 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.y smithi103 *:9283,8765 running (4m) 7s ago 4m 500M - 19.0.0-1423-ge1b79d5a 28480620e5ed 10570ae595fe 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.z smithi164 *:8443,8765 running (2m) 8s ago 2m 449M - 19.0.0-1423-ge1b79d5a 28480620e5ed b95698c4aa63 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:mon.a smithi103 running (4m) 7s ago 4m 47.8M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed cd0bedb0a109 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:mon.b smithi158 running (2m) 8s ago 2m 36.0M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed cd1f74efc812 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:mon.c smithi164 running (3m) 8s ago 3m 36.8M 2048M 19.0.0-1423-ge1b79d5a 28480620e5ed d2117af84018 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (2m) 7s ago 2m 51.0M 4096M 19.0.0-1423-ge1b79d5a 28480620e5ed 29796f451d23 2024-02-13T14:50:50.558 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi158 running (116s) 8s ago 115s 61.5M 4096M 19.0.0-1423-ge1b79d5a 28480620e5ed bae90a4c591d 2024-02-13T14:50:50.559 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi164 running (89s) 8s ago 89s 61.0M 4096M 19.0.0-1423-ge1b79d5a 28480620e5ed fe90ad43a40f 2024-02-13T14:50:50.559 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.a smithi158 *:9095 running (46s) 8s ago 46s 25.9M - 2.43.0 a07b618ecd1d 6ce093964089 2024-02-13T14:50:50.559 INFO:teuthology.orchestra.run.smithi103.stdout:rgw.foo.a smithi103 *:80 running (68s) 7s ago 68s 76.8M - 19.0.0-1423-ge1b79d5a 28480620e5ed 38f145a99eeb 2024-02-13T14:50:50.737 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:50 smithi103 ceph-mon[25851]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:50.737 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:50 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.738 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:50 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.738 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:50 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.738 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:50 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.738 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:50 smithi103 ceph-mon[25851]: Deploying daemon node-exporter.a on smithi103 2024-02-13T14:50:50.834 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-mon[27628]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:50.834 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.834 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.834 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.835 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.835 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-mon[27628]: Deploying daemon node-exporter.a on smithi103 2024-02-13T14:50:50.835 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=infra.usagestats.collector t=2024-02-13T14:50:50.569023436Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-02-13T14:50:50.835 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=provisioning.datasources t=2024-02-13T14:50:50.570124589Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-13T14:50:50.835 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=provisioning.datasources t=2024-02-13T14:50:50.586504497Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-02-13T14:50:50.835 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=provisioning.alerting t=2024-02-13T14:50:50.603789971Z level=info msg="starting to provision alerting" 2024-02-13T14:50:50.835 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=provisioning.alerting t=2024-02-13T14:50:50.603817806Z level=info msg="finished to provision alerting" 2024-02-13T14:50:50.835 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=ngalert.state.manager t=2024-02-13T14:50:50.603874446Z level=info msg="Warming state cache for startup" 2024-02-13T14:50:50.835 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=ngalert.state.manager t=2024-02-13T14:50:50.604049659Z level=info msg="State cache has been initialized" states=0 duration=173.713µs 2024-02-13T14:50:50.835 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=ticker t=2024-02-13T14:50:50.604127239Z level=info msg=starting first_tick=2024-02-13T14:51:00Z 2024-02-13T14:50:50.835 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=ngalert.multiorg.alertmanager t=2024-02-13T14:50:50.604152595Z level=info msg="starting MultiOrg Alertmanager" 2024-02-13T14:50:50.835 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=grafanaStorageLogger t=2024-02-13T14:50:50.604208569Z level=info msg="storage starting" 2024-02-13T14:50:50.836 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=http.server t=2024-02-13T14:50:50.606142938Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-02-13T14:50:50.836 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:50:50 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a[38240]: logger=sqlstore.transactions t=2024-02-13T14:50:50.629581716Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-13T14:50:50.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:50 smithi164 ceph-mon[27636]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:50.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:50 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:50 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:50 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:50 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:50.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:50 smithi164 ceph-mon[27636]: Deploying daemon node-exporter.a on smithi103 2024-02-13T14:50:51.257 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-13T14:50:51.512 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:51.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:51 smithi103 bash[50929]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-13T14:50:51.785 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:51 smithi103 ceph-mon[25851]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:50:51.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:51 smithi164 ceph-mon[27636]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:50:51.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:51 smithi158 ceph-mon[27628]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:50:52.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:52 smithi103 bash[50929]: Getting image source signatures 2024-02-13T14:50:52.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:52 smithi103 bash[50929]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-13T14:50:52.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:52 smithi103 bash[50929]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-13T14:50:52.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:52 smithi103 bash[50929]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-13T14:50:52.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:52 smithi103 ceph-mon[25851]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:52.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:52 smithi164 ceph-mon[27636]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:52.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:52 smithi158 ceph-mon[27628]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:53.934 INFO:teuthology.orchestra.run.smithi103.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-13T14:50:53.934 INFO:teuthology.orchestra.run.smithi103.stdout:agent 3/3 11s ago 41s * 2024-02-13T14:50:53.934 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager ?:9093,9094 1/1 10s ago 66s smithi103=a;count:1 2024-02-13T14:50:53.935 INFO:teuthology.orchestra.run.smithi103.stdout:grafana ?:3000 0/1 - 63s smithi158=a;count:1 2024-02-13T14:50:53.935 INFO:teuthology.orchestra.run.smithi103.stdout:mgr 3/3 11s ago 2m smithi103=y;smithi158=x;smithi164=z;count:3 2024-02-13T14:50:53.935 INFO:teuthology.orchestra.run.smithi103.stdout:mon 3/3 11s ago 3m smithi103:172.21.15.103=a;smithi158:172.21.15.158=b;smithi164:172.21.15.164=c;count:3 2024-02-13T14:50:53.935 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter ?:9100 0/3 - 69s smithi103=a;smithi158=b;smithi164=c;count:3 2024-02-13T14:50:53.935 INFO:teuthology.orchestra.run.smithi103.stdout:osd 3 11s ago - 2024-02-13T14:50:53.935 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus ?:9095 1/1 11s ago 71s smithi158=a;count:1 2024-02-13T14:50:53.935 INFO:teuthology.orchestra.run.smithi103.stdout:rgw.foo.a ?:80 1/1 10s ago 71s smithi103=foo.a;count:1 2024-02-13T14:50:53.952 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:53 smithi103 ceph-mon[25851]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:53.952 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:53 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:54.104 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:53 smithi158 ceph-mon[27628]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:54.104 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:53 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:54.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:53 smithi164 ceph-mon[27636]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:54.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:53 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:55.405 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:50:55 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ::ffff:172.21.15.158 - - [13/Feb/2024:14:50:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-13T14:50:55.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:55 smithi103 ceph-mon[25851]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:50:55.406 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:55 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:55.406 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:55 smithi103 bash[50929]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-13T14:50:55.406 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:55 smithi103 bash[50929]: Writing manifest to image destination 2024-02-13T14:50:55.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:55 smithi164 ceph-mon[27636]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:50:55.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:55 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:55.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:55 smithi158 ceph-mon[27628]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:50:55.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:55 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:55.824 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:55 smithi103 podman[50929]: 2024-02-13 14:50:55.567885498 +0000 UTC m=+4.195428116 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-13T14:50:55.852 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-13T14:50:56.148 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 podman[50929]: 2024-02-13 14:50:56.00582562 +0000 UTC m=+4.633368219 container create e50ab5ac631c220cc6f01f96ef5ed93a567ab1b20cb851d630c13736c872427c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T14:50:56.371 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:56.510 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-mon[25851]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:56.511 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:56.511 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:56.511 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:56.511 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 podman[50929]: 2024-02-13 14:50:56.249872382 +0000 UTC m=+4.877414981 container init e50ab5ac631c220cc6f01f96ef5ed93a567ab1b20cb851d630c13736c872427c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T14:50:56.511 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 podman[50929]: 2024-02-13 14:50:56.255733829 +0000 UTC m=+4.883276431 container start e50ab5ac631c220cc6f01f96ef5ed93a567ab1b20cb851d630c13736c872427c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T14:50:56.511 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.259Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.259Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.259Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.259Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=arp 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-13T14:50:56.512 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=edac 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-13T14:50:56.513 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-13T14:50:56.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-13T14:50:56.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-13T14:50:56.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-13T14:50:56.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-13T14:50:56.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=os 2024-02-13T14:50:56.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-13T14:50:56.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-13T14:50:56.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-13T14:50:56.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-13T14:50:56.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-13T14:50:56.514 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-13T14:50:56.515 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-13T14:50:56.515 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=stat 2024-02-13T14:50:56.515 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-13T14:50:56.515 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-13T14:50:56.515 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-13T14:50:56.515 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=time 2024-02-13T14:50:56.515 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-13T14:50:56.515 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=uname 2024-02-13T14:50:56.515 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-13T14:50:56.515 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-13T14:50:56.515 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-13T14:50:56.516 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-13T14:50:56.516 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a[51259]: ts=2024-02-13T14:50:56.260Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-13T14:50:56.516 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 bash[50929]: e50ab5ac631c220cc6f01f96ef5ed93a567ab1b20cb851d630c13736c872427c 2024-02-13T14:50:56.516 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:50:56 smithi103 systemd[1]: Started Ceph node-exporter.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:50:56.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:56 smithi164 ceph-mon[27636]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:56.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:56 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:56.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:56 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:56.640 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:56 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:56.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:56 smithi158 ceph-mon[27628]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:56.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:56 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:56.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:56 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:56.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:56 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:57.600 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:57 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:57.601 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:57 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:57.601 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:57 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:57.601 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:57 smithi158 ceph-mon[27628]: Deploying daemon node-exporter.b on smithi158 2024-02-13T14:50:57.601 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:50:57 smithi158 systemd[1]: Starting Ceph node-exporter.b for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:50:57.650 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:57 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:57.650 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:57 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:57.650 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:57 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:57.650 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:57 smithi103 ceph-mon[25851]: Deploying daemon node-exporter.b on smithi158 2024-02-13T14:50:57.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:57 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:57.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:57 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:57.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:57 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:50:57.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:57 smithi164 ceph-mon[27636]: Deploying daemon node-exporter.b on smithi158 2024-02-13T14:50:57.928 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:50:57 smithi158 bash[39259]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-13T14:50:58.414 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:50:58 smithi158 bash[39259]: Getting image source signatures 2024-02-13T14:50:58.414 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:50:58 smithi158 bash[39259]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-13T14:50:58.414 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:50:58 smithi158 bash[39259]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-13T14:50:58.414 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:50:58 smithi158 bash[39259]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-13T14:50:58.628 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:58 smithi103 ceph-mon[25851]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:58.628 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:50:58 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:58.638 INFO:teuthology.orchestra.run.smithi103.stdout:HOST ADDR LABELS STATUS 2024-02-13T14:50:58.638 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103 172.21.15.103 2024-02-13T14:50:58.638 INFO:teuthology.orchestra.run.smithi103.stdout:smithi158 172.21.15.158 2024-02-13T14:50:58.638 INFO:teuthology.orchestra.run.smithi103.stdout:smithi164 172.21.15.164 2024-02-13T14:50:58.638 INFO:teuthology.orchestra.run.smithi103.stdout:3 hosts in cluster 2024-02-13T14:50:58.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:58 smithi158 ceph-mon[27628]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:58.679 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:50:58 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:58.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:58 smithi164 ceph-mon[27636]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:50:58.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:50:58 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-13T14:50:59.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-13T14:50:59.639 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:50:59.928 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:50:59 smithi158 bash[39259]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-13T14:50:59.928 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:50:59 smithi158 bash[39259]: Writing manifest to image destination 2024-02-13T14:51:00.417 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 podman[39259]: 2024-02-13 14:51:00.056752855 +0000 UTC m=+2.549955920 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-13T14:51:00.417 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 podman[39259]: 2024-02-13 14:51:00.146683197 +0000 UTC m=+2.639886252 container create 804ee7dd9e670b3fdb10d8ab0f3f65e910731bdf138c6be905cd0ac389994f2d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T14:51:00.417 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 podman[39259]: 2024-02-13 14:51:00.340194211 +0000 UTC m=+2.833397266 container init 804ee7dd9e670b3fdb10d8ab0f3f65e910731bdf138c6be905cd0ac389994f2d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T14:51:00.417 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 podman[39259]: 2024-02-13 14:51:00.346086341 +0000 UTC m=+2.839289397 container start 804ee7dd9e670b3fdb10d8ab0f3f65e910731bdf138c6be905cd0ac389994f2d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T14:51:00.418 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.350Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-13T14:51:00.418 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.350Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-13T14:51:00.418 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-13T14:51:00.418 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-13T14:51:00.418 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-13T14:51:00.418 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-13T14:51:00.418 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-13T14:51:00.418 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=arp 2024-02-13T14:51:00.418 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-13T14:51:00.418 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-13T14:51:00.418 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-13T14:51:00.419 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-13T14:51:00.419 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-13T14:51:00.419 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-13T14:51:00.419 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-13T14:51:00.419 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-13T14:51:00.419 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=edac 2024-02-13T14:51:00.419 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-13T14:51:00.419 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-13T14:51:00.419 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-13T14:51:00.419 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=os 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-13T14:51:00.420 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=stat 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=time 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=uname 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.351Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.352Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-13T14:51:00.421 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b[39310]: ts=2024-02-13T14:51:00.352Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-13T14:51:00.671 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:00 smithi103 ceph-mon[25851]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:51:00.671 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:00 smithi103 ceph-mon[25851]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:00.678 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 bash[39259]: 804ee7dd9e670b3fdb10d8ab0f3f65e910731bdf138c6be905cd0ac389994f2d 2024-02-13T14:51:00.678 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:51:00 smithi158 systemd[1]: Started Ceph node-exporter.b for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:51:00.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-mon[27628]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:51:00.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:00 smithi158 ceph-mon[27628]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:00.845 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:00 smithi164 ceph-mon[27636]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:51:00.845 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:00 smithi164 ceph-mon[27636]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:01.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:01 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:01.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:01 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:01.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:01 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:01.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:01 smithi103 ceph-mon[25851]: Deploying daemon node-exporter.c on smithi164 2024-02-13T14:51:01.775 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:01 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:01.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:01 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:01.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:01 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:01.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:01 smithi164 ceph-mon[27636]: Deploying daemon node-exporter.c on smithi164 2024-02-13T14:51:01.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:01 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:01.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:01 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:01.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:01 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:01.929 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:01 smithi158 ceph-mon[27628]: Deploying daemon node-exporter.c on smithi164 2024-02-13T14:51:02.419 INFO:teuthology.orchestra.run.smithi103.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-13T14:51:02.419 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204012Z400BGN 372G No 19s ago Has a FileSystem, LVM detected 2024-02-13T14:51:02.419 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103 /dev/nvme1n1 ssd Linux_c4a1dd63a4557ea2e6bd 89.4G Yes 19s ago 2024-02-13T14:51:02.419 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103 /dev/nvme2n1 ssd Linux_4908552730dcbfec68b8 89.4G Yes 19s ago 2024-02-13T14:51:02.419 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103 /dev/nvme3n1 ssd Linux_9032664ad82a2742b35e 89.4G Yes 19s ago 2024-02-13T14:51:02.419 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103 /dev/nvme4n1 ssd Linux_cb2988e88cbbcf4aae46 89.4G No 19s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-13T14:51:02.419 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5QBMV 931G No 19s ago Has partitions 2024-02-13T14:51:02.419 INFO:teuthology.orchestra.run.smithi103.stdout:smithi158 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7475005A480DGN 447G No 6s ago Has a FileSystem, LVM detected 2024-02-13T14:51:02.419 INFO:teuthology.orchestra.run.smithi103.stdout:smithi158 /dev/nvme1n1 ssd Linux_f90377fd846a70c9952a 89.4G Yes 6s ago 2024-02-13T14:51:02.419 INFO:teuthology.orchestra.run.smithi103.stdout:smithi158 /dev/nvme2n1 ssd Linux_3a68e5adbaaceb9ca825 89.4G Yes 6s ago 2024-02-13T14:51:02.420 INFO:teuthology.orchestra.run.smithi103.stdout:smithi158 /dev/nvme3n1 ssd Linux_4ee90acd9a7beb69896c 89.4G Yes 6s ago 2024-02-13T14:51:02.420 INFO:teuthology.orchestra.run.smithi103.stdout:smithi158 /dev/nvme4n1 ssd Linux_8c520c98f6908f743af2 89.4G No 6s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-13T14:51:02.420 INFO:teuthology.orchestra.run.smithi103.stdout:smithi158 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XLCL 931G No 6s ago Has partitions 2024-02-13T14:51:02.420 INFO:teuthology.orchestra.run.smithi103.stdout:smithi164 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300H0400BGN 372G No 0s ago Has a FileSystem, LVM detected 2024-02-13T14:51:02.420 INFO:teuthology.orchestra.run.smithi103.stdout:smithi164 /dev/nvme1n1 ssd Linux_1795a1abbc97420e2726 89.4G Yes 0s ago 2024-02-13T14:51:02.420 INFO:teuthology.orchestra.run.smithi103.stdout:smithi164 /dev/nvme2n1 ssd Linux_29182662158a706d639c 89.4G Yes 0s ago 2024-02-13T14:51:02.420 INFO:teuthology.orchestra.run.smithi103.stdout:smithi164 /dev/nvme3n1 ssd Linux_f6fa62dbc83273c92ea1 89.4G Yes 0s ago 2024-02-13T14:51:02.420 INFO:teuthology.orchestra.run.smithi103.stdout:smithi164 /dev/nvme4n1 ssd Linux_abdda64967cbee926047 89.4G No 0s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-13T14:51:02.420 INFO:teuthology.orchestra.run.smithi103.stdout:smithi164 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XTQ7 931G No 0s ago Has partitions 2024-02-13T14:51:02.736 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:02 smithi164 ceph-mon[27636]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:02.736 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:02 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:02.736 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:02 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:02.736 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:02 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:02.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:02 smithi103 ceph-mon[25851]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:02.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:02 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:02.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:02 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:02.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:02 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:02.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:02 smithi158 ceph-mon[27628]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:02.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:02 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:02.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:02 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:02.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:02 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:03.223 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-02-13T14:51:03.503 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:51:03.617 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:03 smithi164 ceph-mon[27636]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:51:03.822 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:03 smithi103 ceph-mon[25851]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:51:03.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:03 smithi158 ceph-mon[27628]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:51:04.390 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:04 smithi164 bash[39835]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-13T14:51:04.890 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:04 smithi164 bash[39835]: Getting image source signatures 2024-02-13T14:51:04.890 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:04 smithi164 bash[39835]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-13T14:51:04.890 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:04 smithi164 bash[39835]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-13T14:51:04.890 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:04 smithi164 bash[39835]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-13T14:51:04.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:04 smithi164 ceph-mon[27636]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:04.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:04 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:04.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:04 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:04.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:04 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:04.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:04 smithi103 ceph-mon[25851]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:04.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:04 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:04.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:04 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:04.905 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:04 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:04.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:04 smithi158 ceph-mon[27628]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:04.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:04 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:04.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:04 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:04.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:04 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:05.404 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:05 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ::ffff:172.21.15.158 - - [13/Feb/2024:14:51:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-13T14:51:05.975 INFO:teuthology.orchestra.run.smithi103.stdout:service_type: agent 2024-02-13T14:51:05.975 INFO:teuthology.orchestra.run.smithi103.stdout:service_name: agent 2024-02-13T14:51:05.975 INFO:teuthology.orchestra.run.smithi103.stdout:placement: 2024-02-13T14:51:05.975 INFO:teuthology.orchestra.run.smithi103.stdout: host_pattern: '*' 2024-02-13T14:51:05.975 INFO:teuthology.orchestra.run.smithi103.stdout:status: 2024-02-13T14:51:05.975 INFO:teuthology.orchestra.run.smithi103.stdout: created: '2024-02-13T14:50:12.761290Z' 2024-02-13T14:51:05.975 INFO:teuthology.orchestra.run.smithi103.stdout: last_refresh: '2024-02-13T14:50:55.766434Z' 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout: running: 3 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout: size: 3 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout:events: 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout:- 2024-02-13T14:50:14.151561Z service:agent [INFO] "service was created" 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout:--- 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout:service_type: alertmanager 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout:service_name: alertmanager 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout:placement: 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout: count: 1 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout: hosts: 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi103=a 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout:status: 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout: created: '2024-02-13T14:49:47.270622Z' 2024-02-13T14:51:05.976 INFO:teuthology.orchestra.run.smithi103.stdout: last_refresh: '2024-02-13T14:51:04.998649Z' 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout: ports: 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout: - 9093 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout: - 9094 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout: running: 1 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout: size: 1 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout:events: 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout:- 2024-02-13T14:50:33.743340Z service:alertmanager [INFO] "service was created" 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout:--- 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout:service_type: grafana 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout:service_name: grafana 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout:placement: 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout: count: 1 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout: hosts: 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi158=a 2024-02-13T14:51:05.977 INFO:teuthology.orchestra.run.smithi103.stdout:spec: 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout: anonymous_access: true 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout: protocol: https 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout:status: 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout: created: '2024-02-13T14:49:49.991023Z' 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout: last_refresh: '2024-02-13T14:50:55.766847Z' 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout: ports: 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout: - 3000 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout: running: 1 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout: size: 1 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout:events: 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout:- 2024-02-13T14:50:49.529154Z service:grafana [INFO] "service was created" 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout:--- 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout:service_type: mgr 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout:service_name: mgr 2024-02-13T14:51:05.978 INFO:teuthology.orchestra.run.smithi103.stdout:placement: 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout: count: 3 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout: hosts: 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi103=y 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi158=x 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi164=z 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout:status: 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout: created: '2024-02-13T14:48:05.231530Z' 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout: last_refresh: '2024-02-13T14:50:55.766618Z' 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout: running: 3 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout: size: 3 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout:--- 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout:service_type: mon 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout:service_name: mon 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout:placement: 2024-02-13T14:51:05.979 INFO:teuthology.orchestra.run.smithi103.stdout: count: 3 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout: hosts: 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi103:172.21.15.103=a 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi158:172.21.15.158=b 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi164:172.21.15.164=c 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout:status: 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout: created: '2024-02-13T14:47:38.109480Z' 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout: last_refresh: '2024-02-13T14:50:55.766537Z' 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout: running: 3 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout: size: 3 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout:--- 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout:service_type: node-exporter 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout:service_name: node-exporter 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout:placement: 2024-02-13T14:51:05.980 INFO:teuthology.orchestra.run.smithi103.stdout: count: 3 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout: hosts: 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi103=a 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi158=b 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi164=c 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout:status: 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout: created: '2024-02-13T14:49:44.672456Z' 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout: ports: 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout: - 9100 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout: running: 1 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout: size: 3 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout:events: 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout:- 2024-02-13T14:51:00.537853Z service:node-exporter [INFO] "service was created" 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout:--- 2024-02-13T14:51:05.981 INFO:teuthology.orchestra.run.smithi103.stdout:service_type: osd 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout:service_name: osd 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout:unmanaged: true 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout:spec: 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout: filter_logic: AND 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout: objectstore: bluestore 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout:status: 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout: container_image_id: 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout: last_refresh: '2024-02-13T14:50:55.766696Z' 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout: running: 3 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout: size: 3 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout:--- 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout:service_type: prometheus 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout:service_name: prometheus 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout:placement: 2024-02-13T14:51:05.982 INFO:teuthology.orchestra.run.smithi103.stdout: count: 1 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout: hosts: 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi158=a 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout:status: 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout: created: '2024-02-13T14:49:41.964500Z' 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout: last_refresh: '2024-02-13T14:50:55.766773Z' 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout: ports: 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout: - 9095 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout: running: 1 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout: size: 1 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout:--- 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout:service_type: rgw 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout:service_id: foo.a 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout:service_name: rgw.foo.a 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout:placement: 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout: count: 1 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout: hosts: 2024-02-13T14:51:05.983 INFO:teuthology.orchestra.run.smithi103.stdout: - smithi103=foo.a 2024-02-13T14:51:05.984 INFO:teuthology.orchestra.run.smithi103.stdout:status: 2024-02-13T14:51:05.984 INFO:teuthology.orchestra.run.smithi103.stdout: created: '2024-02-13T14:49:42.413695Z' 2024-02-13T14:51:05.984 INFO:teuthology.orchestra.run.smithi103.stdout: last_refresh: '2024-02-13T14:51:04.998573Z' 2024-02-13T14:51:05.984 INFO:teuthology.orchestra.run.smithi103.stdout: ports: 2024-02-13T14:51:05.984 INFO:teuthology.orchestra.run.smithi103.stdout: - 80 2024-02-13T14:51:05.984 INFO:teuthology.orchestra.run.smithi103.stdout: running: 1 2024-02-13T14:51:05.984 INFO:teuthology.orchestra.run.smithi103.stdout: size: 1 2024-02-13T14:51:06.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:06 smithi103 ceph-mon[25851]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:06.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:06 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:06.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:06 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:06.272 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:06 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:06.273 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 bash[39835]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-13T14:51:06.273 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 bash[39835]: Writing manifest to image destination 2024-02-13T14:51:06.273 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-mon[27636]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:06.273 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:06.273 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:06.273 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:06.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:06 smithi158 ceph-mon[27628]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:06.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:06 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:06.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:06 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:06.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:06 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:06.697 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-02-13T14:51:06.786 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 podman[39835]: 2024-02-13 14:51:06.523993494 +0000 UTC m=+2.513154395 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-13T14:51:06.786 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 podman[39835]: 2024-02-13 14:51:06.647546567 +0000 UTC m=+2.636707459 container create 993255a1bf08d0c894873d582d46df8d418e7041b74c9bd1eb6ec68dca9c63de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-13T14:51:06.937 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/mon.a/config 2024-02-13T14:51:07.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:07 smithi103 ceph-mon[25851]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-13T14:51:07.390 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:07 smithi164 ceph-mon[27636]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-13T14:51:07.390 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 podman[39835]: 2024-02-13 14:51:06.916016259 +0000 UTC m=+2.905177164 container init 993255a1bf08d0c894873d582d46df8d418e7041b74c9bd1eb6ec68dca9c63de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 podman[39835]: 2024-02-13 14:51:06.921489276 +0000 UTC m=+2.910650170 container start 993255a1bf08d0c894873d582d46df8d418e7041b74c9bd1eb6ec68dca9c63de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.925Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.925Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.926Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.926Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.926Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.926Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=arp 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-13T14:51:07.391 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-13T14:51:07.392 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-13T14:51:07.392 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-13T14:51:07.392 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-13T14:51:07.392 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=edac 2024-02-13T14:51:07.392 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-13T14:51:07.392 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-13T14:51:07.392 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-13T14:51:07.393 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=os 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=stat 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=time 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=uname 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-13T14:51:07.394 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-13T14:51:07.395 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-13T14:51:07.395 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-13T14:51:07.395 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:06 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c[39896]: ts=2024-02-13T14:51:06.927Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-13T14:51:07.395 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:07 smithi164 bash[39835]: 993255a1bf08d0c894873d582d46df8d418e7041b74c9bd1eb6ec68dca9c63de 2024-02-13T14:51:07.395 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:51:07 smithi164 systemd[1]: Started Ceph node-exporter.c for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:51:07.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:07 smithi158 ceph-mon[27628]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:51:08.107 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T14:51:08.108 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: Reconfiguring daemon alertmanager.a on smithi103 2024-02-13T14:51:08.108 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:08 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:08.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:51:08.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:51:08.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:51:08.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:51:08.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T14:51:08.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: Reconfiguring daemon alertmanager.a on smithi103 2024-02-13T14:51:08.179 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:08 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:08.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-13T14:51:08.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-13T14:51:08.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-13T14:51:08.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:08.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: Metadata not up to date on all hosts. Skipping non agent specs 2024-02-13T14:51:08.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-13T14:51:08.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: Reconfiguring daemon alertmanager.a on smithi103 2024-02-13T14:51:08.320 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:08 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:09.417 INFO:teuthology.orchestra.run.smithi103.stdout:osd 3 13s ago - 2024-02-13T14:51:10.488 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:10 smithi103 ceph-mon[25851]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:10.488 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:10 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:10.488 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:10 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:10.488 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:10 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:10.489 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:10 smithi103 ceph-mon[25851]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:51:10.515 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-13T14:51:10.524 INFO:tasks.cephadm:Teardown begin 2024-02-13T14:51:10.524 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T14:51:10.555 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T14:51:10.587 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T14:51:10.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:10 smithi164 ceph-mon[27636]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:10.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:10 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:10.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:10 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:10.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:10 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:10.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:10 smithi164 ceph-mon[27636]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:51:10.618 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-13T14:51:10.618 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-13T14:51:10.633 DEBUG:teuthology.orchestra.run.smithi158:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-13T14:51:10.648 DEBUG:teuthology.orchestra.run.smithi164:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-13T14:51:10.673 INFO:tasks.cephadm:Stopping all daemons... 2024-02-13T14:51:10.673 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-13T14:51:10.673 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.a 2024-02-13T14:51:10.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:10 smithi158 ceph-mon[27628]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-13T14:51:10.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:10 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:10.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:10 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:10.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:10 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:10.678 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:10 smithi158 ceph-mon[27628]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-13T14:51:10.754 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:10 smithi103 systemd[1]: Stopping Ceph mon.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:11.655 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:11 smithi103 ceph-mon[25851]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:11.664 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:11 smithi158 ceph-mon[27628]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:11.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:11 smithi164 ceph-mon[27636]: from='mgr.14430 172.21.15.103:0/2495524980' entity='mgr.y' 2024-02-13T14:51:12.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a[25827]: 2024-02-13T14:51:11.999+0000 7f96f1203700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T14:51:12.405 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a[25827]: 2024-02-13T14:51:11.999+0000 7f96f1203700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-13T14:51:12.405 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:12] ENGINE Bus STOPPING 2024-02-13T14:51:12.405 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T14:51:12.405 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:12] ENGINE Bus STOPPED 2024-02-13T14:51:12.405 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:12] ENGINE Bus STARTING 2024-02-13T14:51:12.405 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:12] ENGINE Serving on http://:::9283 2024-02-13T14:51:12.405 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:12 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:12] ENGINE Bus STARTED 2024-02-13T14:51:12.725 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:12 smithi103 podman[53125]: 2024-02-13 14:51:12.423043829 +0000 UTC m=+0.905392269 container died cd0bedb0a109720fb060602593c8db4ae97a2cfda756aa6e467c02e9408a3dbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T14:51:12.725 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:12 smithi103 podman[53125]: 2024-02-13 14:51:12.723314929 +0000 UTC m=+1.205663368 container cleanup cd0bedb0a109720fb060602593c8db4ae97a2cfda756aa6e467c02e9408a3dbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T14:51:13.071 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:12 smithi103 bash[53125]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a 2024-02-13T14:51:13.071 INFO:journalctl@ceph.mon.a.smithi103.stdout:Feb 13 14:51:12 smithi103 podman[53197]: 2024-02-13 14:51:12.966546598 +0000 UTC m=+0.722230975 container remove cd0bedb0a109720fb060602593c8db4ae97a2cfda756aa6e467c02e9408a3dbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T14:51:13.641 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.a.service' 2024-02-13T14:51:13.688 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:13 smithi103 systemd[1]: Stopping Ceph alertmanager.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:13.689 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T14:51:13.689 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-13T14:51:13.690 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-02-13T14:51:13.690 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.b 2024-02-13T14:51:14.067 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:13 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[48910]: ts=2024-02-13T14:51:13.817Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-13T14:51:14.067 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:13 smithi103 podman[53309]: 2024-02-13 14:51:13.837662149 +0000 UTC m=+0.140998361 container died 23bbc8342d9c4311e1f2e9898d2f0303ec052b814499ba647315e6c95b770791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:51:14.178 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:13 smithi158 systemd[1]: Stopping Ceph mon.b for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:14.401 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:14 smithi103 podman[53309]: 2024-02-13 14:51:14.156226151 +0000 UTC m=+0.459562359 container cleanup 23bbc8342d9c4311e1f2e9898d2f0303ec052b814499ba647315e6c95b770791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:51:14.401 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:14 smithi103 bash[53309]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a 2024-02-13T14:51:14.401 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:14 smithi103 podman[53328]: 2024-02-13 14:51:14.306941731 +0000 UTC m=+0.471921206 container remove 23bbc8342d9c4311e1f2e9898d2f0303ec052b814499ba647315e6c95b770791 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:51:14.655 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:14 smithi103 podman[53328]: 2024-02-13 14:51:14.399501073 +0000 UTC m=+0.564480559 volume remove 302107df868a9e76fee4c6fb45f1d0257764db5cd21636966a49c6974710e4e6 2024-02-13T14:51:14.655 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:14 smithi103 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-13T14:51:14.655 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:14 smithi103 systemd[1]: Stopped Ceph alertmanager.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:51:14.655 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:14 smithi103 systemd[1]: Starting Ceph alertmanager.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:15.065 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:14 smithi103 podman[53428]: 2024-02-13 14:51:14.808326625 +0000 UTC m=+0.090527170 volume create 0858ceee3543d3051c202f43afbb9bbaa4c0ecfc2be32ab96b53f6b5ad225564 2024-02-13T14:51:15.065 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:14 smithi103 podman[53428]: 2024-02-13 14:51:14.73495122 +0000 UTC m=+0.017151777 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-13T14:51:15.066 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:14 smithi103 podman[53428]: 2024-02-13 14:51:14.883584671 +0000 UTC m=+0.165785213 container create 8509f411f2bca1e7b3d487a897497ed2b9e9550f87f9407ecf60af1bb8e32426 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:51:15.404 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: ::ffff:172.21.15.158 - - [13/Feb/2024:14:51:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-13T14:51:15.405 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 podman[53428]: 2024-02-13 14:51:15.169008314 +0000 UTC m=+0.451208856 container init 8509f411f2bca1e7b3d487a897497ed2b9e9550f87f9407ecf60af1bb8e32426 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:51:15.405 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 podman[53428]: 2024-02-13 14:51:15.174428179 +0000 UTC m=+0.456628730 container start 8509f411f2bca1e7b3d487a897497ed2b9e9550f87f9407ecf60af1bb8e32426 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:51:15.405 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[53442]: ts=2024-02-13T14:51:15.185Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-13T14:51:15.405 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[53442]: ts=2024-02-13T14:51:15.185Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-13T14:51:15.405 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[53442]: ts=2024-02-13T14:51:15.187Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.103 port=9094 2024-02-13T14:51:15.405 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[53442]: ts=2024-02-13T14:51:15.188Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-13T14:51:15.405 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[53442]: ts=2024-02-13T14:51:15.211Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T14:51:15.405 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[53442]: ts=2024-02-13T14:51:15.211Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-13T14:51:15.405 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[53442]: ts=2024-02-13T14:51:15.214Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-13T14:51:15.406 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[53442]: ts=2024-02-13T14:51:15.214Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-13T14:51:15.406 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 bash[53428]: 8509f411f2bca1e7b3d487a897497ed2b9e9550f87f9407ecf60af1bb8e32426 2024-02-13T14:51:15.406 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:15 smithi103 systemd[1]: Started Ceph alertmanager.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:51:17.405 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:17 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[53442]: ts=2024-02-13T14:51:17.188Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000585115s 2024-02-13T14:51:18.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:18 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-b[27605]: 2024-02-13T14:51:18.157+0000 7f7547868700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T14:51:18.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:18 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-b[27605]: 2024-02-13T14:51:18.157+0000 7f7547868700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-13T14:51:18.428 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:18 smithi158 podman[39971]: 2024-02-13 14:51:18.270107919 +0000 UTC m=+0.429420627 container died cd1f74efc812685ac576d27bf6410246a64c1421ee9aa0df6d6f26119b587f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-b, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T14:51:18.655 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:18 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:18] ENGINE Bus STOPPING 2024-02-13T14:51:18.655 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:18 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-13T14:51:18.655 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:18 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:18] ENGINE Bus STOPPED 2024-02-13T14:51:18.655 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:18 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:18] ENGINE Bus STARTING 2024-02-13T14:51:18.655 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:18 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:18] ENGINE Serving on http://:::9283 2024-02-13T14:51:18.655 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:18 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: [13/Feb/2024:14:51:18] ENGINE Bus STARTED 2024-02-13T14:51:18.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:18 smithi158 podman[39971]: 2024-02-13 14:51:18.605231814 +0000 UTC m=+0.764544522 container cleanup cd1f74efc812685ac576d27bf6410246a64c1421ee9aa0df6d6f26119b587f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0) 2024-02-13T14:51:18.928 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:18 smithi158 bash[39971]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-b 2024-02-13T14:51:19.384 INFO:journalctl@ceph.mon.b.smithi158.stdout:Feb 13 14:51:19 smithi158 podman[40038]: 2024-02-13 14:51:19.124823498 +0000 UTC m=+0.855068991 container remove cd1f74efc812685ac576d27bf6410246a64c1421ee9aa0df6d6f26119b587f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2) 2024-02-13T14:51:19.389 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.b.service' 2024-02-13T14:51:19.459 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T14:51:19.459 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-02-13T14:51:19.459 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-02-13T14:51:19.459 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.c 2024-02-13T14:51:19.768 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:19 smithi164 systemd[1]: Stopping Ceph mon.c for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:20.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:19 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-c[27600]: 2024-02-13T14:51:19.765+0000 7f9085240700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T14:51:20.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:19 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-c[27600]: 2024-02-13T14:51:19.765+0000 7f9085240700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-13T14:51:20.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:19 smithi164 podman[40603]: 2024-02-13 14:51:19.899612453 +0000 UTC m=+0.244599385 container died d2117af84018745b8725684c15641e485c18474a616719fe65b9d75a2fc0066c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T14:51:20.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:20 smithi164 podman[40603]: 2024-02-13 14:51:20.254827484 +0000 UTC m=+0.599814415 container cleanup d2117af84018745b8725684c15641e485c18474a616719fe65b9d75a2fc0066c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-13T14:51:20.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Feb 13 14:51:20 smithi164 bash[40603]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mon-c 2024-02-13T14:51:20.725 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mon.c.service' 2024-02-13T14:51:20.776 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T14:51:20.776 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-02-13T14:51:20.777 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-13T14:51:20.777 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.y 2024-02-13T14:51:21.074 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:20 smithi103 systemd[1]: Stopping Ceph mgr.y for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:21.355 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:21 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:51:21.070+0000 7f568b20b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T14:51:21.355 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:21 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y[26106]: 2024-02-13T14:51:21.070+0000 7f568b20b700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-13T14:51:21.355 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:21 smithi103 podman[53677]: 2024-02-13 14:51:21.103606021 +0000 UTC m=+0.128937687 container died 10570ae595fe0a0fc95c66cd2741187d33d8a4e69c5c71abd88f72fbf20749ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-13T14:51:21.655 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:21 smithi103 podman[53677]: 2024-02-13 14:51:21.477652556 +0000 UTC m=+0.502984226 container cleanup 10570ae595fe0a0fc95c66cd2741187d33d8a4e69c5c71abd88f72fbf20749ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e1b79d5a4193a484176863ab96c0ca28c00e1662, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-13T14:51:21.655 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Feb 13 14:51:21 smithi103 bash[53677]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-y 2024-02-13T14:51:21.888 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.y.service' 2024-02-13T14:51:21.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T14:51:21.926 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-13T14:51:21.926 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-13T14:51:21.926 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.x 2024-02-13T14:51:22.252 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:51:21 smithi158 systemd[1]: Stopping Ceph mgr.x for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:22.579 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:51:22 smithi158 podman[40343]: 2024-02-13 14:51:22.250140848 +0000 UTC m=+0.132669600 container died 1f155c69d49ca9ed60e0adcf25d3a041a41ed926c194500fd99f1f2f465b436d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-13T14:51:22.854 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:51:22 smithi158 podman[40343]: 2024-02-13 14:51:22.660289995 +0000 UTC m=+0.542818752 container cleanup 1f155c69d49ca9ed60e0adcf25d3a041a41ed926c194500fd99f1f2f465b436d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-13T14:51:22.855 INFO:journalctl@ceph.mgr.x.smithi158.stdout:Feb 13 14:51:22 smithi158 bash[40343]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-x 2024-02-13T14:51:22.989 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.x.service' 2024-02-13T14:51:23.026 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T14:51:23.026 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-13T14:51:23.026 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-02-13T14:51:23.026 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.z 2024-02-13T14:51:23.342 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:51:23 smithi164 systemd[1]: Stopping Ceph mgr.z for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:23.640 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:51:23 smithi164 podman[40732]: 2024-02-13 14:51:23.340837744 +0000 UTC m=+0.120974582 container died b95698c4aa63e65fa1fce70dfce55f58e78af337a17449a46b3e383ebf0fb131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-13T14:51:24.060 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:51:23 smithi164 podman[40732]: 2024-02-13 14:51:23.658351334 +0000 UTC m=+0.438488215 container cleanup b95698c4aa63e65fa1fce70dfce55f58e78af337a17449a46b3e383ebf0fb131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-13T14:51:24.060 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:51:23 smithi164 bash[40732]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z 2024-02-13T14:51:24.060 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:51:23 smithi164 podman[40750]: 2024-02-13 14:51:23.859722351 +0000 UTC m=+0.515113804 container remove b95698c4aa63e65fa1fce70dfce55f58e78af337a17449a46b3e383ebf0fb131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-mgr-z, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-13T14:51:24.060 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Feb 13 14:51:23 smithi164 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-02-13T14:51:24.081 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@mgr.z.service' 2024-02-13T14:51:24.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T14:51:24.118 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-02-13T14:51:24.118 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-13T14:51:24.118 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@osd.0 2024-02-13T14:51:24.404 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:24 smithi103 systemd[1]: Stopping Ceph osd.0 for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:24.904 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:24 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0[38276]: 2024-02-13T14:51:24.425+0000 7f444217d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T14:51:24.904 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:24 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0[38276]: 2024-02-13T14:51:24.425+0000 7f444217d700 -1 osd.0 31 *** Got signal Terminated *** 2024-02-13T14:51:24.905 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:24 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0[38276]: 2024-02-13T14:51:24.425+0000 7f444217d700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T14:51:25.524 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:51:25 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[53442]: ts=2024-02-13T14:51:25.190Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002667567s 2024-02-13T14:51:30.010 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:29 smithi103 podman[53833]: 2024-02-13 14:51:29.7546224 +0000 UTC m=+5.441178555 container died 29796f451d232b91ad70c2dbd515a66abfcf00a2fb02cf79a4011fc35ecf72ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-13T14:51:30.376 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:30 smithi103 podman[53833]: 2024-02-13 14:51:30.157121041 +0000 UTC m=+5.843677239 container cleanup 29796f451d232b91ad70c2dbd515a66abfcf00a2fb02cf79a4011fc35ecf72ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-02-13T14:51:30.377 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:30 smithi103 bash[53833]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0 2024-02-13T14:51:30.655 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:30 smithi103 podman[54324]: 2024-02-13 14:51:30.375035058 +0000 UTC m=+0.622316839 container remove 29796f451d232b91ad70c2dbd515a66abfcf00a2fb02cf79a4011fc35ecf72ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-13T14:51:31.062 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:30 smithi103 podman[54419]: 2024-02-13 14:51:30.667375335 +0000 UTC m=+0.086254167 container create 9ab8fb15405ae8f8f492e3357a56750d8550b27de963b82aa2ea1fbe935d7dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T14:51:31.063 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:30 smithi103 podman[54419]: 2024-02-13 14:51:30.599187743 +0000 UTC m=+0.018066573 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f 2024-02-13T14:51:31.063 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:30 smithi103 podman[54419]: 2024-02-13 14:51:30.893082027 +0000 UTC m=+0.311960856 container init 9ab8fb15405ae8f8f492e3357a56750d8550b27de963b82aa2ea1fbe935d7dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-13T14:51:31.063 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:30 smithi103 podman[54419]: 2024-02-13 14:51:30.901462731 +0000 UTC m=+0.320341569 container start 9ab8fb15405ae8f8f492e3357a56750d8550b27de963b82aa2ea1fbe935d7dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T14:51:31.063 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:31 smithi103 podman[54419]: 2024-02-13 14:51:31.060981277 +0000 UTC m=+0.479860142 container attach 9ab8fb15405ae8f8f492e3357a56750d8550b27de963b82aa2ea1fbe935d7dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-13T14:51:31.316 INFO:journalctl@ceph.osd.0.smithi103.stdout:Feb 13 14:51:31 smithi103 podman[54419]: 2024-02-13 14:51:31.297477334 +0000 UTC m=+0.716356164 container died 9ab8fb15405ae8f8f492e3357a56750d8550b27de963b82aa2ea1fbe935d7dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5) 2024-02-13T14:51:33.063 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@osd.0.service' 2024-02-13T14:51:33.104 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T14:51:33.104 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-13T14:51:33.104 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-13T14:51:33.105 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@osd.1 2024-02-13T14:51:33.428 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:33 smithi158 systemd[1]: Stopping Ceph osd.1 for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:33.928 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:33 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1[32994]: 2024-02-13T14:51:33.437+0000 7ff7e0ba6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T14:51:33.928 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:33 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1[32994]: 2024-02-13T14:51:33.437+0000 7ff7e0ba6700 -1 osd.1 31 *** Got signal Terminated *** 2024-02-13T14:51:33.928 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:33 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1[32994]: 2024-02-13T14:51:33.437+0000 7ff7e0ba6700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T14:51:39.114 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:38 smithi158 podman[40656]: 2024-02-13 14:51:38.851407491 +0000 UTC m=+5.545757436 container died bae90a4c591d70af72e5a39f5ed018115695f1c81a3778c81c5e5cb94deb9f32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-13T14:51:39.428 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:39 smithi158 podman[40656]: 2024-02-13 14:51:39.203184079 +0000 UTC m=+5.897534021 container cleanup bae90a4c591d70af72e5a39f5ed018115695f1c81a3778c81c5e5cb94deb9f32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-13T14:51:39.428 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:39 smithi158 bash[40656]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1 2024-02-13T14:51:39.814 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:39 smithi158 podman[41191]: 2024-02-13 14:51:39.47905598 +0000 UTC m=+0.629949272 container remove bae90a4c591d70af72e5a39f5ed018115695f1c81a3778c81c5e5cb94deb9f32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T14:51:40.117 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:39 smithi158 podman[41289]: 2024-02-13 14:51:39.813302583 +0000 UTC m=+0.108953468 container create fc03d34a0c2ee1fa885ae4102c48acc4cf87de1e7b966b117691396a88dac267 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-13T14:51:40.118 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:39 smithi158 podman[41289]: 2024-02-13 14:51:39.722312033 +0000 UTC m=+0.017962883 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f 2024-02-13T14:51:40.118 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:40 smithi158 podman[41289]: 2024-02-13 14:51:40.032504251 +0000 UTC m=+0.328155090 container init fc03d34a0c2ee1fa885ae4102c48acc4cf87de1e7b966b117691396a88dac267 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-13T14:51:40.118 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:40 smithi158 podman[41289]: 2024-02-13 14:51:40.03870635 +0000 UTC m=+0.334357193 container start fc03d34a0c2ee1fa885ae4102c48acc4cf87de1e7b966b117691396a88dac267 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-13T14:51:40.118 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:40 smithi158 podman[41289]: 2024-02-13 14:51:40.115749368 +0000 UTC m=+0.411400236 container attach fc03d34a0c2ee1fa885ae4102c48acc4cf87de1e7b966b117691396a88dac267 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T14:51:40.428 INFO:journalctl@ceph.osd.1.smithi158.stdout:Feb 13 14:51:40 smithi158 podman[41289]: 2024-02-13 14:51:40.326814216 +0000 UTC m=+0.622465048 container died fc03d34a0c2ee1fa885ae4102c48acc4cf87de1e7b966b117691396a88dac267 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T14:51:40.995 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@osd.1.service' 2024-02-13T14:51:41.039 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T14:51:41.039 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-13T14:51:41.040 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-13T14:51:41.040 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@osd.2 2024-02-13T14:51:41.350 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:41 smithi164 systemd[1]: Stopping Ceph osd.2 for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:41.640 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:41 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2[34235]: 2024-02-13T14:51:41.347+0000 7f4596c05700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T14:51:41.641 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:41 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2[34235]: 2024-02-13T14:51:41.347+0000 7f4596c05700 -1 osd.2 31 *** Got signal Terminated *** 2024-02-13T14:51:41.641 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:41 smithi164 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2[34235]: 2024-02-13T14:51:41.347+0000 7f4596c05700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-13T14:51:47.140 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:46 smithi164 podman[41425]: 2024-02-13 14:51:46.880454566 +0000 UTC m=+5.640839628 container died fe90ad43a40f47bdc683b3e0c010a143c9f3ae3e8a1eb8afd45f9ec383a60734 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-13T14:51:47.390 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:47 smithi164 podman[41425]: 2024-02-13 14:51:47.236028568 +0000 UTC m=+5.996413629 container cleanup fe90ad43a40f47bdc683b3e0c010a143c9f3ae3e8a1eb8afd45f9ec383a60734 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-13T14:51:47.390 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:47 smithi164 bash[41425]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2 2024-02-13T14:51:47.830 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:47 smithi164 podman[41460]: 2024-02-13 14:51:47.403365573 +0000 UTC m=+0.514946665 container remove fe90ad43a40f47bdc683b3e0c010a143c9f3ae3e8a1eb8afd45f9ec383a60734 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T14:51:48.099 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:47 smithi164 podman[41553]: 2024-02-13 14:51:47.82925397 +0000 UTC m=+0.102096129 container create 236ded55891b31c3a6fa869e4d185890158af17e0b743876f0f96cd1700172d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-13T14:51:48.099 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:47 smithi164 podman[41553]: 2024-02-13 14:51:47.745553748 +0000 UTC m=+0.018395913 image pull 28480620e5edd298d55c0e6673dcbad9ed23edd37269028a11063c1b8925676a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f 2024-02-13T14:51:48.099 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:48 smithi164 podman[41553]: 2024-02-13 14:51:48.006849138 +0000 UTC m=+0.279691309 container init 236ded55891b31c3a6fa869e4d185890158af17e0b743876f0f96cd1700172d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-13T14:51:48.100 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:48 smithi164 podman[41553]: 2024-02-13 14:51:48.012836826 +0000 UTC m=+0.285678986 container start 236ded55891b31c3a6fa869e4d185890158af17e0b743876f0f96cd1700172d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T14:51:48.100 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:48 smithi164 podman[41553]: 2024-02-13 14:51:48.097821506 +0000 UTC m=+0.370663670 container attach 236ded55891b31c3a6fa869e4d185890158af17e0b743876f0f96cd1700172d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2-deactivate, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-13T14:51:48.391 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:48 smithi164 podman[41553]: 2024-02-13 14:51:48.309001597 +0000 UTC m=+0.581843769 container died 236ded55891b31c3a6fa869e4d185890158af17e0b743876f0f96cd1700172d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-13T14:51:48.888 INFO:journalctl@ceph.osd.2.smithi164.stdout:Feb 13 14:51:48 smithi164 podman[41599]: 2024-02-13 14:51:48.867489385 +0000 UTC m=+0.553688547 container remove 236ded55891b31c3a6fa869e4d185890158af17e0b743876f0f96cd1700172d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-osd-2-deactivate, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-13T14:51:48.893 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@osd.2.service' 2024-02-13T14:51:48.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T14:51:48.933 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-13T14:51:48.933 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-13T14:51:48.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@rgw.foo.a 2024-02-13T14:51:49.048 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:51:48 smithi103 systemd[1]: Stopping Ceph rgw.foo.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:51:50.337 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:51:50 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-rgw-foo-a[41365]: 2024-02-13T14:51:50.164+0000 7fdc1124a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-13T14:51:50.337 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:51:50 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-rgw-foo-a[41365]: 2024-02-13T14:51:50.164+0000 7fdc1fabbac0 -1 shutting down 2024-02-13T14:52:00.526 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:52:00 smithi103 bash[55326]: time="2024-02-13T14:52:00Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-13T14:52:00.526 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:52:00 smithi103 podman[55326]: 2024-02-13 14:52:00.187883071 +0000 UTC m=+10.217182683 container died 38f145a99eeb1b5c98d186b18b3270ef67a33e510e7cd718919bd8ed9d0f543b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-rgw-foo-a, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-13T14:52:00.526 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:52:00 smithi103 podman[55326]: 2024-02-13 14:52:00.523779009 +0000 UTC m=+10.553078615 container cleanup 38f145a99eeb1b5c98d186b18b3270ef67a33e510e7cd718919bd8ed9d0f543b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-13T14:52:00.526 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:52:00 smithi103 bash[55326]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-rgw-foo-a 2024-02-13T14:52:00.904 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:52:00 smithi103 podman[55450]: 2024-02-13 14:52:00.699921991 +0000 UTC m=+0.508546911 container remove 38f145a99eeb1b5c98d186b18b3270ef67a33e510e7cd718919bd8ed9d0f543b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48116d68f53d3a2ae4d119fcfc94a11cef23cee69b8a381eaa5186380a06fd9f, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-phlogistonjohn-testing-2024-02-12-1654-e1b79d5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-13T14:52:00.905 INFO:journalctl@ceph.rgw.foo.a.smithi103.stdout:Feb 13 14:52:00 smithi103 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-02-13T14:52:00.960 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@rgw.foo.a.service' 2024-02-13T14:52:00.999 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T14:52:00.999 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-13T14:52:01.000 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-13T14:52:01.000 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@prometheus.a 2024-02-13T14:52:01.314 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 systemd[1]: Stopping Ceph prometheus.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:52:01.578 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.312Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-13T14:52:01.578 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.312Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-13T14:52:01.578 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.312Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-13T14:52:01.578 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.312Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-13T14:52:01.578 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.312Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-13T14:52:01.579 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.312Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-13T14:52:01.579 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.312Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-13T14:52:01.579 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.312Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-13T14:52:01.579 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.312Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-13T14:52:01.579 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.313Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-13T14:52:01.579 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.313Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-13T14:52:01.579 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a[35683]: ts=2024-02-13T14:52:01.313Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-13T14:52:01.579 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 podman[42109]: 2024-02-13 14:52:01.322810706 +0000 UTC m=+0.126133120 container died 6ce093964089f942d242ae8c15859302bab98e0d7fd6d9e8f71496f17cdc34a1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T14:52:01.920 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 podman[42109]: 2024-02-13 14:52:01.684375212 +0000 UTC m=+0.487697625 container cleanup 6ce093964089f942d242ae8c15859302bab98e0d7fd6d9e8f71496f17cdc34a1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-13T14:52:01.923 INFO:journalctl@ceph.prometheus.a.smithi158.stdout:Feb 13 14:52:01 smithi158 bash[42109]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-prometheus-a 2024-02-13T14:52:02.051 DEBUG:teuthology.orchestra.run.smithi158:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@prometheus.a.service' 2024-02-13T14:52:02.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-13T14:52:02.089 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-13T14:52:02.089 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-13T14:52:02.281 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:52:03.633 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:52:03 smithi103 systemd[1]: Stopping Ceph alertmanager.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:52:03.904 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:52:03 smithi103 ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a[53442]: ts=2024-02-13T14:52:03.631Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-13T14:52:03.905 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:52:03 smithi103 podman[55827]: 2024-02-13 14:52:03.662299671 +0000 UTC m=+0.139816104 container died 8509f411f2bca1e7b3d487a897497ed2b9e9550f87f9407ecf60af1bb8e32426 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:52:04.239 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:52:04 smithi103 podman[55827]: 2024-02-13 14:52:04.086921599 +0000 UTC m=+0.564438056 container cleanup 8509f411f2bca1e7b3d487a897497ed2b9e9550f87f9407ecf60af1bb8e32426 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:52:04.239 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:52:04 smithi103 bash[55827]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a 2024-02-13T14:52:04.503 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:52:04 smithi103 podman[55846]: 2024-02-13 14:52:04.237908604 +0000 UTC m=+0.567632817 container remove 8509f411f2bca1e7b3d487a897497ed2b9e9550f87f9407ecf60af1bb8e32426 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-13T14:52:04.503 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:52:04 smithi103 podman[55846]: 2024-02-13 14:52:04.296819223 +0000 UTC m=+0.626543436 volume remove 0858ceee3543d3051c202f43afbb9bbaa4c0ecfc2be32ab96b53f6b5ad225564 2024-02-13T14:52:04.905 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:52:04 smithi103 systemd[1]: Stopping Ceph node-exporter.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:52:04.905 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:52:04 smithi103 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-13T14:52:04.905 INFO:journalctl@ceph.alertmanager.a.smithi103.stdout:Feb 13 14:52:04 smithi103 systemd[1]: Stopped Ceph alertmanager.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:52:05.259 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:52:04 smithi103 podman[55965]: 2024-02-13 14:52:04.95485355 +0000 UTC m=+0.112313576 container died e50ab5ac631c220cc6f01f96ef5ed93a567ab1b20cb851d630c13736c872427c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T14:52:05.259 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:52:05 smithi103 podman[55965]: 2024-02-13 14:52:05.257615327 +0000 UTC m=+0.415075337 container cleanup e50ab5ac631c220cc6f01f96ef5ed93a567ab1b20cb851d630c13736c872427c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T14:52:05.654 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:52:05 smithi103 bash[55965]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a 2024-02-13T14:52:05.655 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:52:05 smithi103 podman[55983]: 2024-02-13 14:52:05.46697755 +0000 UTC m=+0.516939147 container remove e50ab5ac631c220cc6f01f96ef5ed93a567ab1b20cb851d630c13736c872427c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-13T14:52:05.655 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:52:05 smithi103 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-13T14:52:06.126 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:52:05 smithi103 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-13T14:52:06.127 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:52:05 smithi103 systemd[1]: Stopped Ceph node-exporter.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:52:06.127 INFO:journalctl@ceph.node-exporter.a.smithi103.stdout:Feb 13 14:52:05 smithi103 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.a.service: Consumed 1.868s CPU time. 2024-02-13T14:52:06.309 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-13T14:52:06.726 INFO:teuthology.orchestra.run.smithi158.stdout:Deleting cluster with fsid: 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:52:08.025 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:52:07 smithi158 systemd[1]: Stopping Ceph grafana.a for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:52:08.288 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:52:08 smithi158 podman[42669]: 2024-02-13 14:52:08.023420358 +0000 UTC m=+0.123395655 container died 9077e6c625be1f6183243007cbb6947b61ad40a15dda8c9227d92a150c334229 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, vcs-type=git, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , description=Ceph Grafana Container, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., version=8.5, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, name=ubi8, release=239.1651231664) 2024-02-13T14:52:08.678 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:52:08 smithi158 podman[42669]: 2024-02-13 14:52:08.359759516 +0000 UTC m=+0.459734813 container cleanup 9077e6c625be1f6183243007cbb6947b61ad40a15dda8c9227d92a150c334229 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, name=ubi8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, architecture=x86_64, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, vcs-type=git, version=8.5, com.redhat.component=ubi8-container, description=Ceph Grafana Container) 2024-02-13T14:52:08.678 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:52:08 smithi158 bash[42669]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a 2024-02-13T14:52:08.678 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:52:08 smithi158 podman[42688]: 2024-02-13 14:52:08.510853124 +0000 UTC m=+0.489242320 container remove 9077e6c625be1f6183243007cbb6947b61ad40a15dda8c9227d92a150c334229 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, description=Ceph Grafana Container, name=ubi8, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., maintainer=Paul Cuzner , io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.tags=base rhel8, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-02-13T14:52:08.678 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:52:08 smithi158 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-02-13T14:52:09.170 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:52:08 smithi158 systemd[1]: Stopping Ceph node-exporter.b for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:52:09.171 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:52:08 smithi158 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-13T14:52:09.171 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:52:08 smithi158 systemd[1]: Stopped Ceph grafana.a for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:52:09.171 INFO:journalctl@ceph.grafana.a.smithi158.stdout:Feb 13 14:52:08 smithi158 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@grafana.a.service: Consumed 3.824s CPU time. 2024-02-13T14:52:09.428 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:52:09 smithi158 podman[42808]: 2024-02-13 14:52:09.169095671 +0000 UTC m=+0.115824598 container died 804ee7dd9e670b3fdb10d8ab0f3f65e910731bdf138c6be905cd0ac389994f2d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T14:52:09.835 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:52:09 smithi158 podman[42808]: 2024-02-13 14:52:09.513980718 +0000 UTC m=+0.460709640 container cleanup 804ee7dd9e670b3fdb10d8ab0f3f65e910731bdf138c6be905cd0ac389994f2d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T14:52:09.835 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:52:09 smithi158 bash[42808]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b 2024-02-13T14:52:09.835 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:52:09 smithi158 podman[42827]: 2024-02-13 14:52:09.639965106 +0000 UTC m=+0.474211828 container remove 804ee7dd9e670b3fdb10d8ab0f3f65e910731bdf138c6be905cd0ac389994f2d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-13T14:52:09.835 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:52:09 smithi158 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-13T14:52:10.104 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:52:09 smithi158 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-13T14:52:10.104 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:52:09 smithi158 systemd[1]: Stopped Ceph node-exporter.b for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:52:10.105 INFO:journalctl@ceph.node-exporter.b.smithi158.stdout:Feb 13 14:52:09 smithi158 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.b.service: Consumed 1.801s CPU time. 2024-02-13T14:52:10.552 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-13T14:52:12.007 INFO:teuthology.orchestra.run.smithi164.stdout:Deleting cluster with fsid: 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:52:13.140 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:52:12 smithi164 systemd[1]: Stopping Ceph node-exporter.c for 7acdb230-ca7e-11ee-95b9-87774f69a715... 2024-02-13T14:52:13.504 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:52:13 smithi164 podman[42669]: 2024-02-13 14:52:13.16458505 +0000 UTC m=+0.106612792 container died 993255a1bf08d0c894873d582d46df8d418e7041b74c9bd1eb6ec68dca9c63de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-13T14:52:13.890 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:52:13 smithi164 podman[42669]: 2024-02-13 14:52:13.592783926 +0000 UTC m=+0.534811671 container cleanup 993255a1bf08d0c894873d582d46df8d418e7041b74c9bd1eb6ec68dca9c63de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-13T14:52:13.890 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:52:13 smithi164 bash[42669]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c 2024-02-13T14:52:13.890 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:52:13 smithi164 podman[42688]: 2024-02-13 14:52:13.743697287 +0000 UTC m=+0.583799534 container remove 993255a1bf08d0c894873d582d46df8d418e7041b74c9bd1eb6ec68dca9c63de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-7acdb230-ca7e-11ee-95b9-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-02-13T14:52:13.890 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:52:13 smithi164 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.c.service: Main process exited, code=exited, status=143/n/a 2024-02-13T14:52:14.213 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:52:13 smithi164 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.c.service: Failed with result 'exit-code'. 2024-02-13T14:52:14.213 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:52:13 smithi164 systemd[1]: Stopped Ceph node-exporter.c for 7acdb230-ca7e-11ee-95b9-87774f69a715. 2024-02-13T14:52:14.213 INFO:journalctl@ceph.node-exporter.c.smithi164.stdout:Feb 13 14:52:13 smithi164 systemd[1]: ceph-7acdb230-ca7e-11ee-95b9-87774f69a715@node-exporter.c.service: Consumed 1.796s CPU time. 2024-02-13T14:52:14.614 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T14:52:14.650 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T14:52:14.686 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-13T14:52:14.715 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-13T14:52:14.717 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066/remote/smithi103/crash 2024-02-13T14:52:14.717 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/crash -- . 2024-02-13T14:52:14.768 INFO:teuthology.orchestra.run.smithi103.stderr:tar: /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-13T14:52:14.768 INFO:teuthology.orchestra.run.smithi103.stderr:tar: Error is not recoverable: exiting now 2024-02-13T14:52:14.770 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066/remote/smithi158/crash 2024-02-13T14:52:14.771 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/crash -- . 2024-02-13T14:52:14.826 INFO:teuthology.orchestra.run.smithi158.stderr:tar: /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-13T14:52:14.826 INFO:teuthology.orchestra.run.smithi158.stderr:tar: Error is not recoverable: exiting now 2024-02-13T14:52:14.827 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066/remote/smithi164/crash 2024-02-13T14:52:14.828 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/crash -- . 2024-02-13T14:52:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:tar: /var/lib/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-13T14:52:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:tar: Error is not recoverable: exiting now 2024-02-13T14:52:14.865 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-13T14:52:14.865 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-13T14:52:14.898 INFO:teuthology.orchestra.run.smithi103.stdout:2024-02-13T14:47:57.436636+0000 mon.a (mon.0) 246 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-13T14:52:14.898 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-13T14:52:14.898 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[SEC\]' /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-13T14:52:14.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[ERR\]' /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-13T14:52:15.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[WRN\]' /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-13T14:52:15.111 INFO:teuthology.orchestra.run.smithi103.stdout:2024-02-13T14:47:57.436636+0000 mon.a (mon.0) 246 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-13T14:52:15.111 INFO:tasks.cephadm:Compressing logs... 2024-02-13T14:52:15.111 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-13T14:52:15.154 DEBUG:teuthology.orchestra.run.smithi158:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-13T14:52:15.157 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-13T14:52:15.181 INFO:teuthology.orchestra.run.smithi103.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-02-13T14:52:15.181 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-02-13T14:52:15.181 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mon.a.log 2024-02-13T14:52:15.181 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-13T14:52:15.183 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-13T14:52:15.183 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mon.a.log: /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mgr.y.log 2024-02-13T14:52:15.183 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-osd.0.log 2024-02-13T14:52:15.183 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-client.rgw.foo.a.log: 57.7% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-13T14:52:15.183 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-13T14:52:15.187 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log 2024-02-13T14:52:15.188 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-13T14:52:15.188 INFO:teuthology.orchestra.run.smithi103.stderr: 89.9% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-13T14:52:15.188 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log: 82.7% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-13T14:52:15.188 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-volume.log: 84.8% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-13T14:52:15.188 INFO:teuthology.orchestra.run.smithi103.stderr: 83.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-13T14:52:15.188 INFO:teuthology.orchestra.run.smithi164.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-02-13T14:52:15.188 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-02-13T14:52:15.189 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-13T14:52:15.189 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mgr.z.log 2024-02-13T14:52:15.189 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-13T14:52:15.189 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log 2024-02-13T14:52:15.189 INFO:teuthology.orchestra.run.smithi164.stderr:gzip/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.cephadm.log: -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-osd.2.log 2024-02-13T14:52:15.189 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log: 81.5% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-13T14:52:15.189 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-13T14:52:15.189 INFO:teuthology.orchestra.run.smithi164.stderr: 90.1% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-13T14:52:15.189 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mon.c.log 2024-02-13T14:52:15.189 INFO:teuthology.orchestra.run.smithi164.stderr: 84.5% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-13T14:52:15.189 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-volume.log: /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mon.c.log: 76.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-13T14:52:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-13T14:52:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-13T14:52:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-13T14:52:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-osd.1.log 2024-02-13T14:52:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mgr.x.log 2024-02-13T14:52:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-13T14:52:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log 2024-02-13T14:52:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.cephadm.log: 90.0% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-13T14:52:15.192 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-13T14:52:15.193 INFO:teuthology.orchestra.run.smithi158.stderr: 81.5% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-13T14:52:15.193 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mon.b.log 2024-02-13T14:52:15.193 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log: /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-volume.log: 84.3% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-13T14:52:15.193 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mon.b.log: 76.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-13T14:52:15.193 INFO:teuthology.orchestra.run.smithi158.stderr: 90.3% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mgr.x.log.gz 2024-02-13T14:52:15.193 INFO:teuthology.orchestra.run.smithi164.stderr: 90.2% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mgr.z.log.gz 2024-02-13T14:52:15.204 INFO:teuthology.orchestra.run.smithi158.stderr: 90.2% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-13T14:52:15.204 INFO:teuthology.orchestra.run.smithi164.stderr: 90.2% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-13T14:52:15.209 INFO:teuthology.orchestra.run.smithi103.stderr: 90.1% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-13T14:52:15.254 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mon.c.log.gz 2024-02-13T14:52:15.261 INFO:teuthology.orchestra.run.smithi158.stderr: 92.4% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mon.b.log.gz 2024-02-13T14:52:15.277 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mgr.y.log.gz 2024-02-13T14:52:15.400 INFO:teuthology.orchestra.run.smithi103.stderr: 91.6% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-mon.a.log.gz 2024-02-13T14:52:15.456 INFO:teuthology.orchestra.run.smithi164.stderr: 94.3% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-osd.2.log.gz 2024-02-13T14:52:15.457 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-02-13T14:52:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m0.286s 2024-02-13T14:52:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:user 0m0.358s 2024-02-13T14:52:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m0.040s 2024-02-13T14:52:15.458 INFO:teuthology.orchestra.run.smithi103.stderr: 94.3% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-osd.0.log.gz 2024-02-13T14:52:15.460 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-02-13T14:52:15.460 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m0.293s 2024-02-13T14:52:15.460 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m0.608s 2024-02-13T14:52:15.460 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.045s 2024-02-13T14:52:15.508 INFO:teuthology.orchestra.run.smithi158.stderr: 94.3% -- replaced with /var/log/ceph/7acdb230-ca7e-11ee-95b9-87774f69a715/ceph-osd.1.log.gz 2024-02-13T14:52:15.510 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-13T14:52:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:real 0m0.339s 2024-02-13T14:52:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:user 0m0.415s 2024-02-13T14:52:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:sys 0m0.041s 2024-02-13T14:52:15.510 INFO:tasks.cephadm:Archiving logs... 2024-02-13T14:52:15.511 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066/remote/smithi103/log 2024-02-13T14:52:15.512 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-13T14:52:15.641 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066/remote/smithi158/log 2024-02-13T14:52:15.642 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-13T14:52:15.730 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066/remote/smithi164/log 2024-02-13T14:52:15.731 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-13T14:52:15.818 INFO:tasks.cephadm:Removing cluster... 2024-02-13T14:52:15.818 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 --force 2024-02-13T14:52:15.997 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:52:16.110 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 --force 2024-02-13T14:52:16.294 INFO:teuthology.orchestra.run.smithi158.stdout:Deleting cluster with fsid: 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:52:16.411 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7acdb230-ca7e-11ee-95b9-87774f69a715 --force 2024-02-13T14:52:16.603 INFO:teuthology.orchestra.run.smithi164.stdout:Deleting cluster with fsid: 7acdb230-ca7e-11ee-95b9-87774f69a715 2024-02-13T14:52:16.721 INFO:tasks.cephadm:Removing cephadm ... 2024-02-13T14:52:16.721 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-13T14:52:16.737 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-13T14:52:16.752 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-13T14:52:16.768 INFO:tasks.cephadm:Teardown complete 2024-02-13T14:52:16.768 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-02-13T14:52:16.778 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi103:/dev/vg_nvme/lv_1... 2024-02-13T14:52:16.778 DEBUG:teuthology.orchestra.run.smithi103:> sudo nvme disconnect -n lv_1 2024-02-13T14:52:16.920 INFO:teuthology.orchestra.run.smithi103.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-02-13T14:52:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:16.922 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi103:/dev/vg_nvme/lv_2... 2024-02-13T14:52:16.922 DEBUG:teuthology.orchestra.run.smithi103:> sudo nvme disconnect -n lv_2 2024-02-13T14:52:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-02-13T14:52:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:17.071 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi103:/dev/vg_nvme/lv_3... 2024-02-13T14:52:17.071 DEBUG:teuthology.orchestra.run.smithi103:> sudo nvme disconnect -n lv_3 2024-02-13T14:52:17.253 INFO:teuthology.orchestra.run.smithi103.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-02-13T14:52:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:17.256 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi103:/dev/vg_nvme/lv_4... 2024-02-13T14:52:17.256 DEBUG:teuthology.orchestra.run.smithi103:> sudo nvme disconnect -n lv_4 2024-02-13T14:52:17.411 INFO:teuthology.orchestra.run.smithi103.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-02-13T14:52:17.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:17.413 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-13T14:52:17.413 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/scratch_devs 2024-02-13T14:52:17.440 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi158:/dev/vg_nvme/lv_1... 2024-02-13T14:52:17.440 DEBUG:teuthology.orchestra.run.smithi158:> sudo nvme disconnect -n lv_1 2024-02-13T14:52:17.574 INFO:teuthology.orchestra.run.smithi158.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-02-13T14:52:17.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:17.576 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi158:/dev/vg_nvme/lv_2... 2024-02-13T14:52:17.576 DEBUG:teuthology.orchestra.run.smithi158:> sudo nvme disconnect -n lv_2 2024-02-13T14:52:17.719 INFO:teuthology.orchestra.run.smithi158.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-02-13T14:52:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:17.720 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi158:/dev/vg_nvme/lv_3... 2024-02-13T14:52:17.721 DEBUG:teuthology.orchestra.run.smithi158:> sudo nvme disconnect -n lv_3 2024-02-13T14:52:17.849 INFO:teuthology.orchestra.run.smithi158.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-02-13T14:52:17.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:17.851 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi158:/dev/vg_nvme/lv_4... 2024-02-13T14:52:17.851 DEBUG:teuthology.orchestra.run.smithi158:> sudo nvme disconnect -n lv_4 2024-02-13T14:52:17.984 INFO:teuthology.orchestra.run.smithi158.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-02-13T14:52:17.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:17.985 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-13T14:52:17.985 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/scratch_devs 2024-02-13T14:52:18.013 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi164:/dev/vg_nvme/lv_1... 2024-02-13T14:52:18.013 DEBUG:teuthology.orchestra.run.smithi164:> sudo nvme disconnect -n lv_1 2024-02-13T14:52:18.166 INFO:teuthology.orchestra.run.smithi164.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-02-13T14:52:18.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:18.168 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi164:/dev/vg_nvme/lv_2... 2024-02-13T14:52:18.168 DEBUG:teuthology.orchestra.run.smithi164:> sudo nvme disconnect -n lv_2 2024-02-13T14:52:18.326 INFO:teuthology.orchestra.run.smithi164.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-02-13T14:52:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:18.328 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi164:/dev/vg_nvme/lv_3... 2024-02-13T14:52:18.328 DEBUG:teuthology.orchestra.run.smithi164:> sudo nvme disconnect -n lv_3 2024-02-13T14:52:18.493 INFO:teuthology.orchestra.run.smithi164.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-02-13T14:52:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:18.495 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi164:/dev/vg_nvme/lv_4... 2024-02-13T14:52:18.495 DEBUG:teuthology.orchestra.run.smithi164:> sudo nvme disconnect -n lv_4 2024-02-13T14:52:18.639 INFO:teuthology.orchestra.run.smithi164.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-02-13T14:52:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:18.641 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-02-13T14:52:18.641 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/scratch_devs 2024-02-13T14:52:18.673 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-13T14:52:18.683 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-13T14:52:18.683 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T14:52:18.686 DEBUG:teuthology.orchestra.run.smithi158:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T14:52:18.689 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-13T14:52:18.699 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-02-13T14:52:18.703 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T14:52:18.703 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-02-13T14:52:18.703 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +434us[ +434us] +/- 91ms 2024-02-13T14:52:18.704 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -1491us[-1492us] +/- 51ms 2024-02-13T14:52:18.704 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +2077us[+2077us] +/- 87ms 2024-02-13T14:52:18.704 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:52:18.704 INFO:teuthology.orchestra.run.smithi158.stderr:bash: line 1: ntpq: command not found 2024-02-13T14:52:18.708 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T14:52:18.708 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2024-02-13T14:52:18.708 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +429us[ +439us] +/- 91ms 2024-02-13T14:52:18.709 INFO:teuthology.orchestra.run.smithi158.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -1484us[-1475us] +/- 51ms 2024-02-13T14:52:18.709 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +2082us[+2091us] +/- 87ms 2024-02-13T14:52:18.709 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:52:18.729 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-02-13T14:52:18.733 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-13T14:52:18.734 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-02-13T14:52:18.734 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +419us[ +428us] +/- 91ms 2024-02-13T14:52:18.734 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -1474us[-1464us] +/- 51ms 2024-02-13T14:52:18.734 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +2091us[+2100us] +/- 87ms 2024-02-13T14:52:18.735 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-13T14:52:18.736 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-13T14:52:18.769 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-13T14:52:18.771 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-13T14:52:18.782 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-13T14:52:18.856 DEBUG:teuthology.orchestra.run.smithi158:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-13T14:52:18.932 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-13T14:52:19.008 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T14:52:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:19.039 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T14:52:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:19.071 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-13T14:52:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:19.105 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-13T14:52:19.115 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-13T14:52:19.123 INFO:teuthology.task.internal:Duration was 697.416310 seconds 2024-02-13T14:52:19.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-13T14:52:19.133 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-13T14:52:19.133 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-13T14:52:19.136 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-13T14:52:19.138 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-13T14:52:19.177 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T14:52:19.181 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T14:52:19.190 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-13T14:52:19.692 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-13T14:52:19.692 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-02-13T14:52:19.692 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-13T14:52:19.753 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi158.front.sepia.ceph.com 2024-02-13T14:52:19.753 DEBUG:teuthology.orchestra.run.smithi158:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-13T14:52:19.792 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2024-02-13T14:52:19.792 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-13T14:52:19.817 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-13T14:52:19.817 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-13T14:52:19.819 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-13T14:52:19.834 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-13T14:52:19.887 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-13T14:52:19.888 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-13T14:52:19.890 DEBUG:teuthology.orchestra.run.smithi158:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-13T14:52:19.908 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-13T14:52:20.085 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-13T14:52:20.095 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-13T14:52:20.095 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-13T14:52:20.122 DEBUG:teuthology.orchestra.run.smithi158:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-13T14:52:20.150 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-13T14:52:20.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-13T14:52:20.187 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-13T14:52:20.190 DEBUG:teuthology.orchestra.run.smithi158:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-13T14:52:20.192 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-13T14:52:20.213 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-02-13T14:52:20.215 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = core 2024-02-13T14:52:20.242 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2024-02-13T14:52:20.267 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-13T14:52:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:20.291 DEBUG:teuthology.orchestra.run.smithi158:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-13T14:52:20.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:20.307 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-13T14:52:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-13T14:52:20.321 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-13T14:52:20.331 INFO:teuthology.task.internal:Transferring archived files... 2024-02-13T14:52:20.332 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066/remote/smithi103 2024-02-13T14:52:20.332 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-13T14:52:20.372 DEBUG:teuthology.misc:Transferring archived files from smithi158:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066/remote/smithi158 2024-02-13T14:52:20.372 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-13T14:52:20.409 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066/remote/smithi164 2024-02-13T14:52:20.409 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-13T14:52:20.446 INFO:teuthology.task.internal:Removing archive directory... 2024-02-13T14:52:20.446 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-13T14:52:20.449 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-13T14:52:20.451 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-13T14:52:20.495 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-13T14:52:20.506 INFO:teuthology.task.internal:Not uploading archives. 2024-02-13T14:52:20.506 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-13T14:52:20.516 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-13T14:52:20.517 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-13T14:52:20.519 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-13T14:52:20.522 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-13T14:52:20.531 INFO:teuthology.orchestra.run.smithi103.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 13 14:52 /home/ubuntu/cephtest 2024-02-13T14:52:20.533 INFO:teuthology.orchestra.run.smithi158.stdout: 265293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 13 14:52 /home/ubuntu/cephtest 2024-02-13T14:52:20.550 INFO:teuthology.orchestra.run.smithi164.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 13 14:52 /home/ubuntu/cephtest 2024-02-13T14:52:20.552 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-13T14:52:20.563 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-13T14:52:20.584 INFO:teuthology.nuke:Checking targets against current locks 2024-02-13T14:52:20.611 DEBUG:teuthology.nuke:shortname: smithi103 2024-02-13T14:52:20.611 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T14:52:20.635 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 14:31:02.183947', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCgujYfr5ZZ2cZPre+/pcryH2hEyo7d4eMgdZPnP5PXcPdOYXdGbRNbxjuSb7MwfEeuxj61WopXC1Jx3WlJ4qjM='} 2024-02-13T14:52:20.639 DEBUG:teuthology.nuke:shortname: smithi158 2024-02-13T14:52:20.639 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T14:52:20.663 DEBUG:teuthology.nuke:shortname: smithi164 2024-02-13T14:52:20.664 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-13T14:52:20.668 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 14:31:02.185540', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIGotlDtCMc9d/iWbkJlYPN8whM0jPDGTzY3QgD11AtZHCDqdkECOfX0jmkf4h7ZvTAak8A25jKg83wLsX6004g='} 2024-02-13T14:52:20.698 INFO:teuthology.orchestra.console.smithi103:Power off 2024-02-13T14:52:20.699 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-13T14:52:20.713 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-02-13_11:41:43-orch:cephadm-wip-phlogistonjohn-testing-2024-02-12-1654-distro-default-smithi/7558066', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-13 14:31:02.186700', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGL5pnV+NJNKGTamqUR0b5k5TOO/txtyKaQ4/YlUqj6BBU753W/lxteVYYkUGml4vzBef1TXJpo1t0sSO2/HaKI='} 2024-02-13T14:52:20.725 DEBUG:teuthology.orchestra.console.smithi103:power off output: Chassis Power Control: Down/Off 2024-02-13T14:52:20.725 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T14:52:20.744 INFO:teuthology.orchestra.console.smithi158:Power off 2024-02-13T14:52:20.744 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-13T14:52:20.755 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-02-13T14:52:20.775 DEBUG:teuthology.orchestra.console.smithi158:power off output: Chassis Power Control: Down/Off 2024-02-13T14:52:20.775 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T14:52:20.796 INFO:teuthology.orchestra.console.smithi164:Power off 2024-02-13T14:52:20.796 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-13T14:52:20.809 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2024-02-13T14:52:20.817 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off 2024-02-13T14:52:20.817 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T14:52:20.840 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-02-13T14:52:24.756 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T14:52:24.810 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T14:52:24.841 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T14:52:24.868 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-02-13T14:52:24.921 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2024-02-13T14:52:24.952 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-02-13T14:52:28.872 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T14:52:28.922 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T14:52:28.953 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T14:52:28.982 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is off 2024-02-13T14:52:29.032 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is off 2024-02-13T14:52:29.062 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-02-13T14:52:29.084 INFO:teuthology.orchestra.console.smithi103:Power off completed 2024-02-13T14:52:29.133 INFO:teuthology.orchestra.console.smithi158:Power off completed 2024-02-13T14:52:33.067 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-13T14:52:33.177 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off 2024-02-13T14:52:33.278 INFO:teuthology.orchestra.console.smithi164:Power off completed 2024-02-13T14:52:33.381 INFO:teuthology.run:Summary data: description: orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} duration: 697.4163098335266 failure_reason: '"2024-02-13T14:47:57.436636+0000 mon.a (mon.0) 246 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_phlogistonjohn@teuthology success: false 2024-02-13T14:52:33.382 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-13T14:52:33.465 INFO:teuthology.run:FAIL